The initialized weights of nn.Linear are very large within __init__
Author: zhjohnchan
Pull request: no

### System Info

Hi,

I wrote a classification wrapper for CLIPModel like this:
```
class CLIPVisionTransformerForImageClassification(CLIPPreTrainedModel):
    config_class = CLIPConfig
    _no_split_modules = ["CLIPEncoderLayer"]

    def __init__(self, config: CLIPConfig):
        super().__init__(config)
        ....
        # Classifier head
        self.classifier = (
            nn.Linear(self.vision_config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
        )
```

In `Transformers==4.35.2`, it works well. After switching to the latest version (i.e., 4.37.2), its initialized weights are very large like
```
self.classifier.weight
Parameter containing:
tensor([[2.7691e+20, 1.7750e+28, 4.1259e-08,  ..., 7.0974e+22, 2.1473e+29,
         1.7861e+25],
        [4.8366e+30, 2.1683e-10, 1.8465e+25,  ..., 2.0505e-10, 1.8612e+34,
         7.6818e+31],
        [1.1319e+21, 2.0683e-10, 1.7034e+25,  ..., 1.1353e+24, 2.8175e+20,
         6.8589e+22],
        [4.5445e+30, 3.0446e+32, 4.3059e+21,  ..., 1.8526e+28, 5.3747e-11,
         1.7077e+22],
        [2.0505e-10, 1.8612e+34, 7.6818e+31,  ..., 3.6291e-09, 6.8906e+22,
         2.6350e+23]], device='cuda:0', requires_grad=True)
```

Any thoughts about this? Thanks!

Best,

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
class CLIPVisionTransformerForImageClassification(CLIPPreTrainedModel):
    config_class = CLIPConfig
    _no_split_modules = ["CLIPEncoderLayer"]

    def __init__(self, config: CLIPConfig):
        super().__init__(config)
        ....
        # Classifier head
        self.classifier = (
            nn.Linear(self.vision_config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
        )
```

### Expected behavior

```
self.classifier.weight
Parameter containing:
tensor([[2.7691e+20, 1.7750e+28, 4.1259e-08,  ..., 7.0974e+22, 2.1473e+29,
         1.7861e+25],
        [4.8366e+30, 2.1683e-10, 1.8465e+25,  ..., 2.0505e-10, 1.8612e+34,
         7.6818e+31],
        [1.1319e+21, 2.0683e-10, 1.7034e+25,  ..., 1.1353e+24, 2.8175e+20,
         6.8589e+22],
        [4.5445e+30, 3.0446e+32, 4.3059e+21,  ..., 1.8526e+28, 5.3747e-11,
         1.7077e+22],
        [2.0505e-10, 1.8612e+34, 7.6818e+31,  ..., 3.6291e-09, 6.8906e+22,
         2.6350e+23]], device='cuda:0', requires_grad=True)
```

Comments:
Looks like a precision problem? The values are in the normal range in version 4.35.2.

cc @ydshieh this is an issue you looked into

Hi @zhjohnchan We do have similar issue, but I am surprised that v4.35 has normal values. I will have to take a look into this.
BTW, could you share you torch versions, especially if the torch version is the same while you switch transformers from  4.35 to 4.37.

Thanks in advance.

Thanks @NielsRogge and @ydshieh,

The following is my torch version:
```
torch                         2.0.1
torchaudio                    2.0.2
torchmetrics                  0.11.4
torchvision                   0.15.2
```

The torch version is fixed in my setting.

Best,


Seems something in Transformers overwrites the operators? I also tried nn.init, it's the same problem.

Hi @zhjohnchan Could you try the following code and see what the results look like?

I am not able to reproduce (Version: 2.0.1+cu118, transformers=4.37.2)

**Question: are you using torch cpu or torch gpu?**

```python
from transformers import CLIPPreTrainedModel, CLIPConfig
import torch
import torch.nn as nn


class CLIPVisionTransformerForImageClassification(CLIPPreTrainedModel):
    config_class = CLIPConfig
    _no_split_modules = ["CLIPEncoderLayer"]

    def __init__(self, config: CLIPConfig):
        super().__init__(config)
        # Classifier head
        self.classifier = (
            nn.Linear(config.vision_config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
        )


config = CLIPConfig()
model = CLIPVisionTransformerForImageClassification(config)
print(model.classifier.weight)
```

### The results I got

```
root@9cfc7e033729:/transformers# python3 temp.py
Parameter containing:
tensor([[-0.0050,  0.0280,  0.0112,  ..., -0.0241, -0.0166, -0.0334],
        [ 0.0351,  0.0027,  0.0015,  ...,  0.0028, -0.0343, -0.0290]],
       requires_grad=True)
tensor(0.0360, grad_fn=<AmaxBackward0>)
```

If the above example has no issue on your side, I will need your **complete** code that could reproduce the issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.