Choice of variable name in custom model affects model initialization
Author: urmeya
Pull request: no
Labels: name: bug

### System Info

transformers version 4.17.0
python version 3.7.11
platform Ubuntu

### Who can help?

@LysandreJik  @sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to build a custom model by using one of the transformer language models as base model and and then defining a custom head. Below is an example code that is working.

```python
from transformers import AutoModel, AutoConfig, PreTrainedModel
from transformers.modeling_outputs import SequenceClassifierOutput
import torch
import torch.nn as nn

class CustomModel(PreTrainedModel):
    def __init__(self, config, num_labels=2, dropout_prob=0.3):
        super(CustomModel, self).__init__(config)
        self.num_labels = num_labels
        self.bert = AutoModel.from_config(config)
        self.dropout = nn.Dropout(dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, num_labels)
        
    def _init_weights(self, module):
        if isinstance(module, (nn.Linear, nn.Embedding)):
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
    
    def forward(self, input_ids, attention_mask, labels=None):
        
        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        # sequence_output = self.dropout(outputs[0])
        pooled_output = outputs[1]
        
        logits = self.classifier(pooled_output)
        
        loss = None
        if labels is not None:
            loss_fct = nn.CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        return SequenceClassifierOutput(loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions)
    
checkpoint = "bert-base-uncased"
config = AutoConfig.from_pretrained(checkpoint)
model = CustomModel.from_pretrained(pretrained_model_name_or_path=checkpoint, config=config, num_labels=2, dropout_prob=0.3)
``` 
Here I have used `bert` as the base model and the variable name is also `self.bert`. I get following warning which I think is okay.
```
Some weights of the model checkpoint at bert-base-uncased were not used when initializing CustomModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing CustomModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CustomModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of CustomModel were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'bert.embeddings.position_ids', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```
The problem arises when I just change variable name `self.bert` to any other name like `self.basemodel` as in following code

```python
from transformers import AutoModel, AutoConfig, PreTrainedModel
from transformers.modeling_outputs import SequenceClassifierOutput
import torch
import torch.nn as nn

class CustomModel(PreTrainedModel):
    def __init__(self, config, num_labels=2, dropout_prob=0.3):
        super(CustomModel, self).__init__(config)
        self.num_labels = num_labels
        self.basemodel = AutoModel.from_config(config)
        self.dropout = nn.Dropout(dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, num_labels)
        
    def _init_weights(self, module):
        if isinstance(module, (nn.Linear, nn.Embedding)):
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
    
    def forward(self, input_ids, attention_mask, labels=None):
        
        outputs = self.basemodel(input_ids=input_ids, attention_mask=attention_mask)
        # sequence_output = self.dropout(outputs[0])
        pooled_output = outputs[1]
        
        logits = self.classifier(pooled_output)
        
        loss = None
        if labels is not None:
            loss_fct = nn.CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        return SequenceClassifierOutput(loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions)
    
checkpoint = "bert-base-uncased"
config = AutoConfig.from_pretrained(checkpoint)
model = CustomModel.from_pretrained(pretrained_model_name_or_path=checkpoint, config=config, num_labels=2, dropout_prob=0.3)
```
Here I get following warning:
```
Some weights of the model checkpoint at bert-base-uncased were not used when initializing CustomModel: ['bert.encoder.layer.6.attention.self.query.weight', 'bert.encoder.layer.1.attention.output.LayerNorm.bias', 'bert.encoder.layer.8.attention.output.LayerNorm.weight', 'bert.encoder.layer.7.attention.output.dense.weight', 'bert.encoder.layer.7.output.LayerNorm.weight', 'bert.encoder.layer.6.output.dense.bias', 'bert.encoder.layer.2.attention.self.query.weight', 'bert.encoder.layer.0.attention.self.value.bias', 'bert.encoder.layer.10.attention.output.LayerNorm.bias', 'bert.encoder.layer.4.output.dense.bias', 'bert.encoder.layer.7.intermediate.dense.weight', 'bert.encoder.layer.0.attention.output.LayerNorm.weight', 'bert.encoder.layer.5.attention.self.query.bias', 'bert.encoder.layer.7.attention.output.LayerNorm.weight', 'bert.encoder.layer.3.attention.self.key.weight', 'bert.encoder.layer.10.attention.self.key.bias', 'bert.encoder.layer.0.attention.self.key.bias', 'bert.encoder.layer.1.attention.self.query.bias', 'bert.encoder.layer.2.attention.self.value.weight', 'bert.encoder.layer.9.attention.self.value.bias', 'bert.encoder.layer.4.output.LayerNorm.weight', 'bert.encoder.layer.1.attention.self.key.bias', 'bert.encoder.layer.11.attention.self.query.bias', 'cls.predictions.decoder.weight', 'bert.encoder.layer.10.output.LayerNorm.bias', 'bert.encoder.layer.3.output.LayerNorm.weight', 'bert.encoder.layer.8.attention.self.value.weight', 'bert.encoder.layer.3.attention.output.LayerNorm.weight', 'bert.encoder.layer.6.output.LayerNorm.weight', 'bert.encoder.layer.10.attention.self.query.bias', 'bert.encoder.layer.8.attention.self.query.weight', 'bert.encoder.layer.0.attention.self.query.bias', 'bert.encoder.layer.1.output.LayerNorm.weight', 'bert.encoder.layer.1.intermediate.dense.weight', 'bert.encoder.layer.8.attention.self.key.weight', 'bert.encoder.layer.6.attention.output.dense.weight', 'bert.encoder.layer.8.attention.output.dense.weight', 'bert.encoder.layer.0.attention.output.dense.weight', 'bert.encoder.layer.0.output.LayerNorm.bias', 'bert.encoder.layer.7.intermediate.dense.bias', 'bert.encoder.layer.8.intermediate.dense.bias', 'bert.encoder.layer.1.output.dense.weight', 'bert.encoder.layer.11.attention.output.dense.weight', 'bert.encoder.layer.9.attention.self.value.weight', 'bert.encoder.layer.1.attention.self.key.weight', 'bert.encoder.layer.0.intermediate.dense.weight', 'bert.encoder.layer.5.attention.self.value.bias', 'bert.encoder.layer.9.attention.self.query.bias', 'bert.encoder.layer.10.intermediate.dense.weight', 'bert.encoder.layer.11.attention.output.dense.bias', 'bert.encoder.layer.9.attention.self.key.bias', 'bert.encoder.layer.3.attention.output.dense.weight', 'bert.encoder.layer.8.intermediate.dense.weight', 'bert.embeddings.LayerNorm.bias', 'bert.encoder.layer.8.output.dense.weight', 'bert.encoder.layer.9.output.dense.weight', 'bert.encoder.layer.7.attention.self.key.weight', 'bert.encoder.layer.11.output.dense.weight', 'bert.encoder.layer.5.attention.self.key.weight', 'bert.encoder.layer.0.output.LayerNorm.weight', 'bert.embeddings.position_embeddings.weight', 'bert.encoder.layer.10.attention.output.dense.weight', 'bert.encoder.layer.9.intermediate.dense.bias', 'bert.encoder.layer.7.attention.self.query.weight', 'bert.encoder.layer.4.output.LayerNorm.bias', 'bert.encoder.layer.7.output.LayerNorm.bias', 'bert.encoder.layer.5.attention.output.LayerNorm.weight', 'bert.encoder.layer.4.attention.output.LayerNorm.bias', 'bert.encoder.layer.6.output.LayerNorm.bias', 'bert.encoder.layer.11.output.LayerNorm.weight', 'bert.encoder.layer.7.attention.self.key.bias', 'bert.encoder.layer.8.attention.output.dense.bias', 'bert.encoder.layer.1.output.dense.bias', 'bert.encoder.layer.4.attention.output.dense.bias', 'bert.encoder.layer.3.output.LayerNorm.bias', 'bert.encoder.layer.4.attention.self.value.bias', 'bert.encoder.layer.3.attention.self.query.weight', 'bert.encoder.layer.7.attention.self.value.weight', 'bert.encoder.layer.8.attention.self.key.bias', 'bert.embeddings.token_type_embeddings.weight', 'bert.encoder.layer.4.attention.self.key.weight', 'bert.encoder.layer.11.attention.output.LayerNorm.weight', 'bert.encoder.layer.3.intermediate.dense.bias', 'bert.encoder.layer.6.attention.self.query.bias', 'bert.encoder.layer.11.attention.self.key.weight', 'bert.encoder.layer.1.intermediate.dense.bias', 'bert.encoder.layer.5.attention.self.key.bias', 'bert.encoder.layer.6.attention.output.LayerNorm.bias', 'bert.encoder.layer.2.attention.output.dense.weight', 'bert.encoder.layer.11.output.LayerNorm.bias', 'bert.encoder.layer.0.attention.self.value.weight', 'cls.predictions.transform.dense.bias', 'bert.encoder.layer.2.attention.output.LayerNorm.bias', 'bert.encoder.layer.0.intermediate.dense.bias', 'bert.encoder.layer.0.attention.output.LayerNorm.bias', 'bert.encoder.layer.9.intermediate.dense.weight', 'bert.encoder.layer.0.attention.self.query.weight', 'bert.encoder.layer.11.attention.output.LayerNorm.bias', 'bert.encoder.layer.6.attention.self.key.bias', 'bert.encoder.layer.11.attention.self.key.bias', 'bert.encoder.layer.5.attention.output.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'bert.encoder.layer.3.attention.self.query.bias', 'bert.encoder.layer.10.attention.output.LayerNorm.weight', 'bert.encoder.layer.6.attention.self.key.weight', 'bert.encoder.layer.10.output.dense.bias', 'bert.encoder.layer.7.attention.output.LayerNorm.bias', 'bert.encoder.layer.8.output.LayerNorm.weight', 'bert.encoder.layer.5.attention.output.dense.weight', 'bert.encoder.layer.10.attention.self.query.weight', 'bert.encoder.layer.6.attention.self.value.weight', 'bert.encoder.layer.10.attention.output.dense.bias', 'bert.encoder.layer.5.intermediate.dense.weight', 'bert.encoder.layer.6.attention.self.value.bias', 'bert.encoder.layer.4.output.dense.weight', 'bert.embeddings.word_embeddings.weight', 'bert.encoder.layer.10.intermediate.dense.bias', 'bert.encoder.layer.7.output.dense.weight', 'bert.encoder.layer.4.attention.output.LayerNorm.weight', 'bert.encoder.layer.5.output.LayerNorm.bias', 'bert.encoder.layer.1.attention.self.value.bias', 'bert.encoder.layer.8.output.dense.bias', 'bert.encoder.layer.9.attention.output.LayerNorm.bias', 'bert.encoder.layer.2.attention.self.query.bias', 'bert.encoder.layer.4.attention.self.value.weight', 'bert.encoder.layer.5.attention.self.query.weight', 'bert.encoder.layer.10.attention.self.value.bias', 'bert.encoder.layer.4.attention.self.key.bias', 'bert.encoder.layer.10.attention.self.value.weight', 'bert.encoder.layer.5.output.dense.bias', 'bert.encoder.layer.2.output.LayerNorm.bias', 'bert.encoder.layer.2.output.dense.weight', 'bert.encoder.layer.11.intermediate.dense.bias', 'bert.encoder.layer.0.output.dense.weight', 'bert.encoder.layer.3.output.dense.weight', 'bert.encoder.layer.4.attention.output.dense.weight', 'bert.encoder.layer.5.intermediate.dense.bias', 'bert.encoder.layer.3.output.dense.bias', 'cls.seq_relationship.weight', 'bert.encoder.layer.5.output.dense.weight', 'bert.encoder.layer.2.intermediate.dense.weight', 'bert.encoder.layer.11.intermediate.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'bert.encoder.layer.11.output.dense.bias', 'cls.predictions.bias', 'bert.encoder.layer.4.intermediate.dense.bias', 'bert.encoder.layer.3.attention.self.value.weight', 'bert.encoder.layer.9.attention.self.key.weight', 'bert.encoder.layer.11.attention.self.value.weight', 'bert.encoder.layer.5.attention.self.value.weight', 'bert.encoder.layer.10.output.LayerNorm.weight', 'bert.encoder.layer.3.intermediate.dense.weight', 'bert.encoder.layer.1.attention.self.value.weight', 'bert.pooler.dense.bias', 'bert.encoder.layer.3.attention.output.dense.bias', 'bert.encoder.layer.8.attention.self.query.bias', 'bert.encoder.layer.8.attention.self.value.bias', 'bert.encoder.layer.10.output.dense.weight', 'bert.encoder.layer.9.attention.output.dense.bias', 'bert.embeddings.LayerNorm.weight', 'bert.encoder.layer.1.attention.output.dense.bias', 'bert.pooler.dense.weight', 'bert.encoder.layer.8.output.LayerNorm.bias', 'bert.encoder.layer.9.attention.output.dense.weight', 'bert.encoder.layer.6.attention.output.LayerNorm.weight', 'bert.encoder.layer.7.attention.self.value.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'bert.encoder.layer.2.attention.self.key.weight', 'bert.encoder.layer.8.attention.output.LayerNorm.bias', 'bert.encoder.layer.7.attention.self.query.bias', 'bert.encoder.layer.10.attention.self.key.weight', 'bert.encoder.layer.6.intermediate.dense.weight', 'bert.encoder.layer.5.output.LayerNorm.weight', 'bert.encoder.layer.2.attention.output.dense.bias', 'bert.encoder.layer.6.attention.output.dense.bias', 'bert.encoder.layer.0.output.dense.bias', 'bert.encoder.layer.0.attention.output.dense.bias', 'bert.encoder.layer.3.attention.self.key.bias', 'bert.encoder.layer.6.output.dense.weight', 'bert.encoder.layer.4.attention.self.query.bias', 'bert.encoder.layer.11.attention.self.value.bias', 'bert.encoder.layer.3.attention.self.value.bias', 'bert.encoder.layer.7.attention.output.dense.bias', 'bert.encoder.layer.2.attention.self.value.bias', 'bert.encoder.layer.1.attention.output.LayerNorm.weight', 'bert.encoder.layer.1.attention.self.query.weight', 'bert.encoder.layer.4.intermediate.dense.weight', 'bert.encoder.layer.9.output.LayerNorm.weight', 'bert.encoder.layer.0.attention.self.key.weight', 'bert.encoder.layer.9.output.dense.bias', 'bert.encoder.layer.9.attention.self.query.weight', 'bert.encoder.layer.4.attention.self.query.weight', 'bert.encoder.layer.6.intermediate.dense.bias', 'bert.encoder.layer.2.attention.output.LayerNorm.weight', 'bert.encoder.layer.2.attention.self.key.bias', 'bert.encoder.layer.1.output.LayerNorm.bias', 'bert.encoder.layer.2.intermediate.dense.bias', 'bert.encoder.layer.7.output.dense.bias', 'bert.encoder.layer.2.output.dense.bias', 'bert.encoder.layer.1.attention.output.dense.weight', 'bert.encoder.layer.5.attention.output.dense.bias', 'bert.encoder.layer.3.attention.output.LayerNorm.bias', 'bert.encoder.layer.2.output.LayerNorm.weight', 'bert.encoder.layer.9.output.LayerNorm.bias', 'bert.encoder.layer.9.attention.output.LayerNorm.weight', 'bert.encoder.layer.11.attention.self.query.weight']
- This IS expected if you are initializing CustomModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CustomModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of CustomModel were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['basemodel.encoder.layer.5.output.dense.bias', 'basemodel.encoder.layer.5.attention.output.dense.weight', 'basemodel.encoder.layer.5.attention.self.query.bias', 'basemodel.pooler.dense.weight', 'basemodel.encoder.layer.9.attention.self.query.bias', 'basemodel.encoder.layer.0.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.10.attention.self.query.weight', 'basemodel.encoder.layer.10.intermediate.dense.weight', 'basemodel.encoder.layer.10.attention.self.key.weight', 'basemodel.encoder.layer.10.attention.output.dense.bias', 'basemodel.encoder.layer.0.attention.output.dense.bias', 'basemodel.encoder.layer.7.intermediate.dense.weight', 'basemodel.encoder.layer.0.output.dense.weight', 'basemodel.encoder.layer.0.attention.self.key.weight', 'basemodel.encoder.layer.3.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.11.attention.self.value.bias', 'basemodel.encoder.layer.0.attention.self.key.bias', 'basemodel.encoder.layer.6.output.dense.weight', 'basemodel.encoder.layer.1.attention.self.query.bias', 'basemodel.encoder.layer.6.attention.output.dense.bias', 'basemodel.encoder.layer.9.attention.self.query.weight', 'basemodel.encoder.layer.1.attention.output.dense.weight', 'basemodel.encoder.layer.8.attention.self.value.weight', 'basemodel.encoder.layer.0.output.dense.bias', 'basemodel.encoder.layer.4.attention.self.value.bias', 'basemodel.encoder.layer.1.attention.self.key.bias', 'basemodel.encoder.layer.5.attention.self.key.bias', 'basemodel.encoder.layer.9.intermediate.dense.bias', 'basemodel.encoder.layer.5.intermediate.dense.bias', 'basemodel.encoder.layer.7.attention.self.value.bias', 'basemodel.encoder.layer.4.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.6.attention.output.dense.weight', 'basemodel.encoder.layer.7.output.dense.bias', 'basemodel.encoder.layer.3.attention.self.query.bias', 'basemodel.encoder.layer.4.attention.output.dense.bias', 'basemodel.encoder.layer.8.attention.self.value.bias', 'basemodel.encoder.layer.0.attention.self.value.bias', 'basemodel.encoder.layer.8.attention.self.query.weight', 'basemodel.encoder.layer.6.intermediate.dense.bias', 'basemodel.encoder.layer.10.output.dense.weight', 'basemodel.encoder.layer.2.attention.self.key.bias', 'basemodel.encoder.layer.5.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.9.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.1.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.10.attention.self.query.bias', 'basemodel.encoder.layer.6.output.LayerNorm.weight', 'basemodel.encoder.layer.11.attention.self.query.weight', 'basemodel.encoder.layer.3.attention.self.value.weight', 'basemodel.encoder.layer.4.output.dense.weight', 'basemodel.encoder.layer.11.attention.self.query.bias', 'basemodel.encoder.layer.6.attention.self.value.weight', 'basemodel.encoder.layer.4.intermediate.dense.weight', 'basemodel.encoder.layer.3.output.dense.weight', 'basemodel.encoder.layer.2.attention.output.dense.weight', 'basemodel.encoder.layer.2.attention.self.query.weight', 'basemodel.encoder.layer.9.attention.output.dense.weight', 'basemodel.encoder.layer.11.intermediate.dense.weight', 'basemodel.encoder.layer.9.attention.self.key.weight', 'basemodel.encoder.layer.10.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.2.output.LayerNorm.bias', 'basemodel.encoder.layer.9.attention.self.value.bias', 'basemodel.encoder.layer.5.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.8.output.dense.weight', 'basemodel.encoder.layer.3.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.5.output.LayerNorm.bias', 'basemodel.encoder.layer.9.attention.self.key.bias', 'basemodel.encoder.layer.7.attention.output.dense.weight', 'basemodel.encoder.layer.4.attention.output.dense.weight', 'basemodel.encoder.layer.2.output.dense.bias', 'basemodel.encoder.layer.9.output.LayerNorm.weight', 'basemodel.encoder.layer.3.attention.output.dense.bias', 'basemodel.encoder.layer.11.attention.self.value.weight', 'basemodel.encoder.layer.0.intermediate.dense.weight', 'basemodel.encoder.layer.3.intermediate.dense.weight', 'basemodel.encoder.layer.6.attention.self.value.bias', 'basemodel.encoder.layer.3.attention.output.dense.weight', 'basemodel.encoder.layer.6.attention.self.query.bias', 'basemodel.encoder.layer.2.attention.self.key.weight', 'basemodel.encoder.layer.5.attention.self.key.weight', 'basemodel.encoder.layer.7.output.dense.weight', 'basemodel.encoder.layer.10.output.dense.bias', 'basemodel.encoder.layer.1.attention.self.key.weight', 'basemodel.embeddings.position_ids', 'basemodel.encoder.layer.2.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.10.output.LayerNorm.weight', 'basemodel.encoder.layer.1.attention.self.value.bias', 'basemodel.encoder.layer.7.attention.self.key.weight', 'basemodel.encoder.layer.6.attention.self.key.weight', 'basemodel.encoder.layer.9.intermediate.dense.weight', 'basemodel.embeddings.LayerNorm.weight', 'basemodel.encoder.layer.2.intermediate.dense.weight', 'basemodel.encoder.layer.8.intermediate.dense.bias', 'basemodel.encoder.layer.4.attention.self.key.bias', 'classifier.bias', 'basemodel.encoder.layer.11.attention.self.key.weight', 'basemodel.encoder.layer.0.attention.self.query.bias', 'basemodel.pooler.dense.bias', 'basemodel.encoder.layer.5.attention.output.dense.bias', 'basemodel.encoder.layer.11.attention.output.dense.bias', 'basemodel.encoder.layer.7.attention.self.value.weight', 'basemodel.encoder.layer.1.attention.self.value.weight', 'basemodel.encoder.layer.0.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.4.attention.self.key.weight', 'basemodel.encoder.layer.6.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.2.intermediate.dense.bias', 'basemodel.encoder.layer.10.attention.output.dense.weight', 'basemodel.encoder.layer.11.intermediate.dense.bias', 'basemodel.encoder.layer.6.attention.self.query.weight', 'basemodel.encoder.layer.8.output.LayerNorm.weight', 'basemodel.encoder.layer.7.attention.self.key.bias', 'basemodel.encoder.layer.0.output.LayerNorm.bias', 'basemodel.encoder.layer.11.attention.self.key.bias', 'basemodel.encoder.layer.5.attention.self.value.weight', 'basemodel.encoder.layer.4.attention.self.query.weight', 'basemodel.encoder.layer.7.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.3.attention.self.key.weight', 'basemodel.encoder.layer.1.output.LayerNorm.weight', 'basemodel.encoder.layer.3.attention.self.key.bias', 'basemodel.encoder.layer.0.output.LayerNorm.weight', 'basemodel.encoder.layer.1.attention.output.dense.bias', 'basemodel.encoder.layer.1.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.6.output.LayerNorm.bias', 'basemodel.encoder.layer.8.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.11.attention.output.LayerNorm.weight', 'classifier.weight', 'basemodel.embeddings.token_type_embeddings.weight', 'basemodel.encoder.layer.9.output.LayerNorm.bias', 'basemodel.encoder.layer.0.intermediate.dense.bias', 'basemodel.encoder.layer.4.output.LayerNorm.weight', 'basemodel.encoder.layer.9.attention.output.dense.bias', 'basemodel.encoder.layer.2.attention.self.query.bias', 'basemodel.encoder.layer.8.output.LayerNorm.bias', 'basemodel.encoder.layer.11.attention.output.LayerNorm.bias', 'basemodel.embeddings.LayerNorm.bias', 'basemodel.encoder.layer.8.intermediate.dense.weight', 'basemodel.encoder.layer.2.attention.self.value.weight', 'basemodel.encoder.layer.6.attention.output.LayerNorm.bias', 'basemodel.encoder.layer.11.output.dense.weight', 'basemodel.encoder.layer.3.output.dense.bias', 'basemodel.encoder.layer.4.attention.self.query.bias', 'basemodel.encoder.layer.3.output.LayerNorm.bias', 'basemodel.encoder.layer.4.output.LayerNorm.bias', 'basemodel.encoder.layer.5.attention.self.query.weight', 'basemodel.encoder.layer.5.output.LayerNorm.weight', 'basemodel.encoder.layer.6.output.dense.bias', 'basemodel.encoder.layer.2.attention.self.value.bias', 'basemodel.encoder.layer.8.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.10.attention.self.value.weight', 'basemodel.encoder.layer.9.attention.self.value.weight', 'basemodel.encoder.layer.3.output.LayerNorm.weight', 'basemodel.encoder.layer.7.output.LayerNorm.bias', 'basemodel.encoder.layer.9.output.dense.bias', 'basemodel.encoder.layer.0.attention.output.dense.weight', 'basemodel.encoder.layer.1.intermediate.dense.bias', 'basemodel.encoder.layer.0.attention.self.query.weight', 'basemodel.encoder.layer.1.output.dense.weight', 'basemodel.encoder.layer.8.attention.self.key.weight', 'basemodel.encoder.layer.9.output.dense.weight', 'basemodel.encoder.layer.11.attention.output.dense.weight', 'basemodel.encoder.layer.4.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.7.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.4.intermediate.dense.bias', 'basemodel.encoder.layer.1.attention.self.query.weight', 'basemodel.encoder.layer.6.intermediate.dense.weight', 'basemodel.encoder.layer.7.intermediate.dense.bias', 'basemodel.encoder.layer.10.output.LayerNorm.bias', 'basemodel.encoder.layer.10.attention.self.key.bias', 'basemodel.encoder.layer.5.intermediate.dense.weight', 'basemodel.encoder.layer.4.output.dense.bias', 'basemodel.encoder.layer.8.attention.output.dense.bias', 'basemodel.encoder.layer.8.attention.output.dense.weight', 'basemodel.encoder.layer.10.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.11.output.dense.bias', 'basemodel.encoder.layer.1.output.dense.bias', 'basemodel.encoder.layer.9.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.1.intermediate.dense.weight', 'basemodel.encoder.layer.7.attention.self.query.weight', 'basemodel.encoder.layer.10.attention.self.value.bias', 'basemodel.embeddings.position_embeddings.weight', 'basemodel.encoder.layer.7.output.LayerNorm.weight', 'basemodel.encoder.layer.2.attention.output.LayerNorm.weight', 'basemodel.encoder.layer.3.attention.self.value.bias', 'basemodel.encoder.layer.0.attention.self.value.weight', 'basemodel.encoder.layer.5.attention.self.value.bias', 'basemodel.encoder.layer.6.attention.self.key.bias', 'basemodel.encoder.layer.2.attention.output.dense.bias', 'basemodel.encoder.layer.5.output.dense.weight', 'basemodel.encoder.layer.4.attention.self.value.weight', 'basemodel.encoder.layer.8.output.dense.bias', 'basemodel.encoder.layer.7.attention.self.query.bias', 'basemodel.encoder.layer.3.attention.self.query.weight', 'basemodel.encoder.layer.7.attention.output.dense.bias', 'basemodel.encoder.layer.11.output.LayerNorm.bias', 'basemodel.encoder.layer.2.output.dense.weight', 'basemodel.encoder.layer.2.output.LayerNorm.weight', 'basemodel.encoder.layer.3.intermediate.dense.bias', 'basemodel.embeddings.word_embeddings.weight', 'basemodel.encoder.layer.11.output.LayerNorm.weight', 'basemodel.encoder.layer.8.attention.self.key.bias', 'basemodel.encoder.layer.1.output.LayerNorm.bias', 'basemodel.encoder.layer.10.intermediate.dense.bias', 'basemodel.encoder.layer.8.attention.self.query.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Because of the prefix `basemodel` instead of `bert` in the keys, the weights are not getting mapped. Is there a way to tell `AutoModel` not to add prefix `bert` in the keys?



### Expected behavior

The main reason I want to use generic variable like `self.basemodel` is that I will like to explore different base models which may not necessarily bert models (and therefore even variable name `self.bert` might fail in those cases).
I was hoping that if I  just change the checkpoint name, I should be able to try out different base models.

While exploring the solution, I found following code snippet that adds/removes `prefix` like `bert` if required. But in my case I would need to first remove prefix `basemodel` and add then add `bert` prefix which not possible in the current and wanyway would be too messy.

https://github.com/huggingface/transformers/blob/855dcae8bb743c3f8f0781742d7fa2fa3aaa3e22/src/transformers/modeling_utils.py#L2321-L2340

Please help me in figuring out a correct way to achieve what I am trying to do.

Comments:
I am very confused as to what the bug you think you have here is. You are trying to load weights of a checkpoint in a model that does not match (`"bert-base-ubcased"` is a BERT model with no head, so it does not expect a `bert` attribute). Using the base model prefix is the magic that Transformers uses behind the scenes to load those checkpoints in model with heads. 

In the two code implementations I have pasted, the only difference is a variable name (`self.bert` in first case and `self.basemodel` in second case).

My query is: if the model keys get mapped correctly in first case, why are they not mapping in second case? 

Please compare the warning messages I have pasted for the two cases if this description is not making sense. 


> My query is: if the model keys get mapped correctly in first case, why are they not mapping in second case?

Your description shows the exact opposite: there is a warning in the first case and not in the second case. Please clarify what it is you are asking as I don't understand your question.

I missed word 'not' there. Correcting it now.

@sgugger and @LysandreJik , please find more concise version of my query below. 

I am trying to write a custom model class that can be used to finetune transformers language model like BERT with custom head for some downstream task.

The code below is my attempt to write such class. It uses `bert-base-uncased` as base model and adds custom layer(s) (just a linear layer for this example case). The code below loads the pretrained `bert-base-uncased` weights for finetuning and also randomly initiates the custom head layer for training. 

Please let me know if it is a correct way to write custom model class for finetuning.
The problem I am facing is that the code below stops working (i.e. doesn't load pretrained weights but randomly initializes all of them) if I just change the variable name `self.bert` to any other name. I want to know if it is an expected behavior. If yes, for any specific model on huggingface hub, how can I find which variable name (similar to `self.bert`) one is supposed to use?

```python
from transformers import AutoModel, AutoConfig, PreTrainedModel
from transformers.modeling_outputs import SequenceClassifierOutput
import torch
import torch.nn as nn

class CustomModel(PreTrainedModel):
    def __init__(self, config, num_labels=2, dropout_prob=0.3):
        super(CustomModel, self).__init__(config)
        self.num_labels = num_labels
        self.bert = AutoModel.from_config(config)
        self.dropout = nn.Dropout(dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, num_labels)
        
    def _init_weights(self, module):
        if isinstance(module, (nn.Linear, nn.Embedding)):
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
    
    def forward(self, input_ids, attention_mask, labels=None):
        
        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        # sequence_output = self.dropout(outputs[0])
        pooled_output = outputs[1]
        
        logits = self.classifier(pooled_output)
        
        loss = None
        if labels is not None:
            loss_fct = nn.CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        return SequenceClassifierOutput(loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions)
    
checkpoint = "bert-base-uncased"
config = AutoConfig.from_pretrained(checkpoint)
model = CustomModel.from_pretrained(pretrained_model_name_or_path=checkpoint, config=config, num_labels=2, dropout_prob=0.3)
```

First note that such a generic class goes against Transformers design principles, which is one class per model. So, it's logical you would have to change the name of the attribute for each model with head you want to write.

Otherwise you can try setting the model using `self.base_model_prefix` (which will be `"bert"` for BERT, "roberta" for RoBERTa etc.) inside your custom model, if you want your class to be generic.

I had tried setting `self.base_model_prefix` but it does not work as it adds prefix on top of variable name (e.g. `bert.basemodel.encoder.layer.9.output.LayerNorm.weight` if the variable name is `self.basemodel`)

I do get your bigger point that using generic class like `PreTrainedModel` is against transformers design principles. I will write the model classes inheriting from specific model classes like `BertPreTrainedModel` / `RobertaPreTrainedModel` and strictly using corresponding variable name like `self.bert`/`self.roberta`.


Hi! I'm facing the same problem. Did you find any solution, @urmeya ?