Why we need the init_weight function in BERT pretrained model
Author: allanj
Pull request: no
Labels: name: Usage

# ❓ Questions & Help

I have already tried asking the question is SO, which you can find the link [here](https://stackoverflow.com/questions/62040309/why-we-need-the-init-weight-function-in-bert-pretrained-model-in-huggingface-tra/62053791#62053791). 

## Details
In the code by Hugginface transformers, there are many fine-tuning models have the function `init_weight`. 
For example([here](https://github.com/huggingface/transformers/blob/a9aa7456ac/src/transformers/modeling_bert.py#L1073-L1082)), there is a `init_weight` function at last. Even though we use `from_pretrained`, it will still call the constructor and call `init_weight` function.
```python
class BertForSequenceClassification(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        self.init_weights()

```
As I know, it will call the following [code](https://github.com/huggingface/transformers/blob/a9aa7456ac/src/transformers/modeling_bert.py#L520-L530)
```python
def _init_weights(self, module):
    """ Initialize the weights """
    if isinstance(module, (nn.Linear, nn.Embedding)):
        # Slightly different from the TF version which uses truncated_normal for initialization
        # cf https://github.com/pytorch/pytorch/pull/5617
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
    elif isinstance(module, BertLayerNorm):
        module.bias.data.zero_()
        module.weight.data.fill_(1.0)
    if isinstance(module, nn.Linear) and module.bias is not None:
        module.bias.data.zero_()
```

My question is **If we are loading the pre-trained language model, why do we need to initialize the weight for every module?**

I guess I must be misunderstanding something here.




Comments:
Have a look at the code for [`.from_pretrained()`](https://github.com/huggingface/transformers/blob/a9aa7456ac824c9027385b149f405e4f5649273f/src/transformers/modeling_utils.py#L490). What actually happens is something like this:

- find the correct base model class to initialise
- initialise that class with pseudo-random initialisation (by using the `_init_weights` function that you mention)
- find the file with the pretrained weights
- overwrite the weights of the model that we just created with the pretrained weightswhere applicable

This ensure that layers were not pretrained (e.g. in some cases the final classification layer) _do_ get initialised in `_init_weights` but don't get overridden.

Great. Thanks. I also read through the code and that really clears my confusion. 

Good. If the answer was sufficient on Stack Overflow as well, please close that too. 





> Have a look at the code for [`.from_pretrained()`](https://github.com/huggingface/transformers/blob/a9aa7456ac824c9027385b149f405e4f5649273f/src/transformers/modeling_utils.py#L490). What actually happens is something like this:
> 
> * find the correct base model class to initialise
> * initialise that class with pseudo-random initialisation (by using the `_init_weights` function that you mention)
> * find the file with the pretrained weights
> * overwrite the weights of the model that we just created with the pretrained weightswhere applicable
> 
> This ensure that layers were not pretrained (e.g. in some cases the final classification layer) _do_ get initialised in `_init_weights` but don't get overridden.

when we construct BertForSequenceClassification from pre-trained model, didn't we overwrite the loaded weights with random initialisation?

@sunersheng No, the random initialization happens [first](https://github.com/huggingface/transformers/blob/a9aa7456ac824c9027385b149f405e4f5649273f/src/transformers/modeling_utils.py#L659) and then the existing weights are loaded [into it](https://github.com/huggingface/transformers/blob/a9aa7456ac824c9027385b149f405e4f5649273f/src/transformers/modeling_utils.py#L732).