`_fast_init` overwrites weights passed to custom model
Author: patrickvonplaten
Pull request: no
Labels: name: Usage

### System Info

```shell
Transformers > 4.6.0
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Copy-pasting an issue from https://github.com/huggingface/transformers/pull/11471#issuecomment-1132324119 into a new issue to make it more visible as multiple people could have this problem.

When abstracting a `transformers` model and passing custom layers to the init method, the initialization can show strange behavior. 

Consider the following case (taken from @josh-heyer ):

```python
from torch.nn import Linear
from transformers import BertModel


class MyCustomModel(BertModel):
    def __init__(self, config, custom_layer=None):
        super().__init__(config)
        if custom_layer is not None:
            self.custom_layer = custom_layer
        else:
            self.custom_layer = Linear(1024, 1024)


if __name__ == "__main__":
    import transformers
    print(transformers.__version__)
    layer = Linear(1024, 1024)
    print(layer.weight.sum())
    custom_model = MyCustomModel.from_pretrained('bert-base-uncased', custom_layer=layer)
    # used to be the same as the layer above, but it is "re-initialized" in the from_pretrained method
    print(custom_model.custom_layer.weight.sum())
```

What will happen here since `_fast_init` was introduced in 4.6.0: https://github.com/huggingface/transformers/pull/11471 is that the weights of the custom layer will be overriden.

### Expected behavior

It might be reasonable to state that the custom layer should **not** be overriden.


Comments:
To describe what happens here:

If `_fast_init=True` (which is set by default) the following happens. All weight names of the `state_dict` loaded from `bert-base-uncased` are compared with all weight names of a random `state_dict` that is generated when calling `MyCustomModel()`. 
Inside `from_pretrained(...)` at this point: https://github.com/huggingface/transformers/blob/7213a40bd914776a6dcebcc96353c4cf8c8c6668/src/transformers/modeling_utils.py#L2284 `custom_layer` is considered a missing layer because it cannot be found in the `state_dict` loaded from `bert-base-uncased` and thus will be randomly initialized, which happens **after** `self.custom_layer` has been set to the passed `layer` weight meaning `self.custom_layer` will be overridden.

This seems unexpected from the outside, but after having looked a bit into the internals of `from_pretrained(...)`, we sadly cannot really change this behavior and also don't consider it a bug.

In our opinion the problems rather lies in the following:

- 1. We **never** abstract `nn.Module` model classes that have an `__init__(...)` method except for the most basic `PretrainedModel` class which has an absolute minimal `__init__(...)` method (see [here](https://github.com/huggingface/transformers/blob/7213a40bd914776a6dcebcc96353c4cf8c8c6668/src/transformers/modeling_utils.py#L980) that just sets the config. 
In the whole code base of `transformers`, models, such as `BertModel`, only abstract from their respective `...PretrainedModel` class, *e.g.* `BertModel` abstracts from `BertPretrainedModel`, but those classes don't have an `__init__` method, see [here]( https://github.com/huggingface/transformers/blob/3fd7de49f4d53832621ef67cfd1825b000323250/src/transformers/models/bert/modeling_bert.py#L733). 

This way we can be sure that the only `__init__(...)` method that matters is the one of `BertModel`. Now this is broken here. `MyCustomModel` abstracts away `BertModel` which is exactly not what we want. 
In short, we **always** favor **modularization over abstraction**. In our opinion, it is less error-prone and easier to understand. 

- 2. We never have want to allow passing layers, such as `nn.Linear` into the `__init__` for the outer-most `nn.Module` classes, not even conditionally. The reason is that, if we allow / recommend this design, it would also mean that one could / should pass a trainable layer through `from_pretrained(...)`. We definitely don't want this as it breaks the assumption that a model is **always** self-contained a single checkpoint, *e.g.* `bert-base-cased` and would therefore make `from_pretrained(...)` very complex. So we never want users to pass trainable layers through `from_pretrained(...)`.



To solve the problem above, we recommend to instead of using abstraction, to just use modularization:


```python
from torch.nn import Linear
from transformers import BertPreTrainedModel, BertModel


class MyCustomModel(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.bert = BertModel(config)
        self.custom_layer = Linear(1024, 1024)

   def forward(self, ...):
       self.bert(....)

   def set_custom_layer(self, linear_embed):
       self.custom_layer = linear_embed
```

Note that this is also exactly how we coded up `BertForMaskedLM`: https://github.com/huggingface/transformers/blob/3fd7de49f4d53832621ef67cfd1825b000323250/src/transformers/models/bert/modeling_bert.py#L1292

Thanks @patrickvonplaten  - btw you tagged "josh-heyer" rather than "john"* haha. I'll take a deeper look and see how we can change our model initialization.

Ups sorry :sweat_smile: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.