[model loading] don't init weights for pretrained models
Author: stas00
Pull request: yes

Skip `_init_weights` for pretrained models since they get immediately replaced by pretrained weights. This leads to a much faster startup for huge models.

Fixes: https://github.com/huggingface/transformers/issues/9205

@sgugger, @patrickvonplaten 

Comments:
> This won't work sadly, for two reasons.
> 
> 1. First, everything in the `__dict__` of the config gets serialized when one uses `config.self_pretrained()` (which is called by `model.from_pretrained`) so any other model downloaded from the hub with a checkpoint saved after this is merged will get this attribute in the config. Then if a user instantiates a randomly-initialized model using the config, with the following code:
> 
> 
> ```python
> config = AutoConfig.from_pretrained("new_checkpoint_after_this_is_merge")
> model = AutoModel.from_config(config)
> ```
> 
> then the model won't be randomly initalized (at least not with `_init_weights`) since the config will have this `use_pretrained_weights`.

So if I find another way to do it that doesn't taint the config then it's OK, right? (as far as config correctness goes)

e.g. what if I unset this config value as soon as `model = cls()` is done? So this is sort of a "context" operation then.

> 2. Then come the problem that pretrained model instantiated with `from_pretrained` does not necessarily have all weights initialized (if you discard the head to put another task-specific head) and this PR will break the way those weights are randomly initialized.
> 
> 
> I sadly don't see a way around passing around a list of not-initialized weights from pretrained to the `_init_weights` function

I appreciate that you could think of the edge cases.

Clearly, we don't have any tests that somehow verify that the init is done correctly. I was hoping that there would be some, but these would be hard to conjure.

If you feel this is a worthwhile effort, perhaps let's start coming up with examples, write tests if possible and solve those? You can throw the edge-cases at me and I will try to overcome those.

Or alternatively, we provide a very easy way for users to either force the init, or if it's safer to force no-init? e.g. the staple examples could all enforce no-init and explain how to change that if the user wants to modify the example to have the original behavior?

So what I'm suggesting is that instead of `from_pretrained` automatically forcing no init as I proposed in this PR, we instead have a way for a user to choose whether they want init_weights or not explicitly?

> I appreciate that you could think of the edge cases.

That is not the edge case but the overwhelming majority ;-) You are mostly working with seq2seq models that don't throw away any weights when doing transfer learning, but all the basic examples fine-tuning BERT on a classification task encounter this :-)

Testing the init is done properly is very difficult as those are all random weights. Testing those weights follow this distribution instead of that one is not something easily achievable.

I don't think the `no_init` option is the right one: it will only work for a certain class of problems and not others, so it's not general enough. We shouldn't go for it just before it's easier to implement than the other solutions on the table.

@stas00  @sgugger that's how I would approach the problem: https://github.com/huggingface/transformers/pull/11471

OK, let's move the effort to Patrick's PR https://github.com/huggingface/transformers/pull/11471

> [...] You are mostly working with seq2seq models [...]

Guilty as charged. I'm glad you guys have a much wider view than I. Thank you!