Why using empty tensor to initialize?
Author: estelleafl
Pull request: no

https://github.com/huggingface/transformers/blob/673440d073d5f534a6d6bedeeca94869afd8d0a7/src/transformers/modeling_utils.py#L3995

Hi,
Why would you use empty tensor to initialize the missing weights? Initializing like so introduces nan for those elements.
Tahnk you 
Best
Estelle

Comments:
Hi @estelleafl, 

Could you provide some more information about when you're experiencing nans, ideally with a reproducible code snippet? It's possible there's a weight init bug in our code which needs addressing. 

With regards to why this is done, it enables faster loaded with pretrained weights. When doing something like the following: 

```py
from transformers import AutoModel

model = AutoModel.from_pretrained(checkpoint)
```

If we don't created empty weights, then all of the model weights are first randomly initialized, and then overwritten with the checkpoints i.e. there's time spent initializing which isn't needed. This way, we just create placeholders which are then filled by the checkpoints. Any weights not included in the checkpoint should then be randomly initialized. 

Thank you @amyeroberts for your fast response

If you do the following

```
import torch
empty_tensor = torch.empty(1000, dtype=torch.float16)
print(empty_tensor.isnan().any())
```

You should see the nan values. I understand now the motivations. But how this issue could be addressed then?

Thank you!


@estelleafl Yes, if one creates tensors like this then you can end up with Nans. torch.empty is just grabbing free memory, so you'll end up with all sorts of values, including nans. 

I don't think there is any issue to address - all of the model's weights should be filled once the class has been instantiated: any weights coming from a pretrained checkpoint will be filled with those values, and any weights which aren't in the pretrained checkpoint will be randomly initialized. 

Hi @amyeroberts  
Sorry for the delay in my answer
If we introduce weights that are not part of the pretrained checkpoint then it can cause issues. Initializing with nan values prevent from backpropagating gradients doesnt it?


Hi @estelleafl, 

Two points about this :

> Initializing with nan values prevent from backpropagating gradients doesnt it?

Using `torch.empty` isn't initializing with `nans`. It's grabbing available memory, which may or may not have nans

> If we introduce weights that are not part of the pretrained checkpoint then it can cause issues. 

I'm not sure what you mean by "introducing weights" here. There are three possible scenarios: 

* Weights which are in the model's architecture but not in the pretrained checkpoint. These will be randomly initialized with float values. 
* A new class is added to the model architecture. Providing this is done properly e.g. like adding a task head, then like above these weights will be randomly initialized. 
* Weights are in the pretrained checkpoint but not in the model definition - these will not be loaded it (not will they be nans)

Perhaps you could share a code snippet which reproduces this issue to clarify? 



Hi @amyeroberts 
Lets say Im using a transformer that I modified a bit by adding another FC layer at the end. I want then to finetune it
When initializing the new trasnformer from pretrained, the weights associated with this new FC layers will be init with this torch empty. So when Ill want to finetune, those weights are going to have some nans. 
What do you think?
Best
Estelle

> Weights which are in the model's architecture but not in the pretrained checkpoint. These will be randomly initialized with float values.

@estelleafl As stated above, this shouldn't be the case. If you add new layers, then these will be part of the architecture that won't have weights in the loaded checkpoint i.e. these "new weights" should be randomly initialized with floats. 

If this isn't the case, could you share a code reproducer in case there's some kind of initialization bug? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.