Custom nn.Parameter initialization in PreTrainedModel subclasses is overwritten by post_init()/from_pretrained() causing NaNs/Zeros
Author: Noietch
Pull request: no
Labels: name: Usage

name: Feature request

name: bug

### System Info

- `transformers` version: 4.57.1
- Platform: Linux-4.18.0-147.mt20200626.413.el8_1.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.35.3
- Safetensors version: 0.6.2
- Accelerate version: 1.11.0
- Accelerate config:    not found
- DeepSpeed version: 0.18.2
- PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA A100-SXM4-80GB


### Who can help?

@Cyrilvallez @zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import numpy as np
import os
import random
import torch
import torch.nn as nn

from transformers import Qwen3VLForConditionalGeneration


def seed_everything(TORCH_SEED):
    random.seed(TORCH_SEED)
    os.environ["PYTHONHASHSEED"] = str(TORCH_SEED)
    np.random.seed(TORCH_SEED)
    torch.manual_seed(TORCH_SEED)
    torch.cuda.manual_seed(TORCH_SEED)
    torch.cuda.manual_seed_all(TORCH_SEED)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False


seed_everything(66)


class TestModel1(Qwen3VLForConditionalGeneration):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.action_head = nn.Linear(1024, 7)
        self.positional_embedding = nn.Parameter(torch.randn(16, 1152))
        self.post_init()


class TestModel2(nn.Module):
    def __init__(self, *args, model_path, **kwargs):
        super().__init__(*args, **kwargs)
        self.model = Qwen3VLForConditionalGeneration.from_pretrained(model_path)
        self.action_head = nn.Linear(1024, 7)
        self.positional_embedding = nn.Parameter(torch.randn(16, 1152))


test_model1 = TestModel1.from_pretrained("Qwen/Qwen3-VL-4B-Instruct")
test_model2 = TestModel2(model_path="Qwen/Qwen3-VL-4B-Instruct")
print(test_model1.positional_embedding)
print(test_model1.positional_embedding.mean(), test_model1.positional_embedding.std())
print(test_model2.positional_embedding)
print(test_model2.positional_embedding.mean(), test_model2.positional_embedding.std())
````

### Expected behavior

When subclassing a model (inheriting from PreTrainedModel, e.g., Qwen3VLForConditionalGeneration, LlamaForCausalLM) to add custom learnable parameters, user-defined initialization in __init__ is often silently overwritten.

This occurs because from_pretrained (or the end of __init__) triggers self.post_init(), which recursively calls _init_weights. This mechanism re-initializes all parameters, ignoring the explicit initialization code provided by the user in __init__.

In the specific case of Qwen3-VL (and potentially others), this re-initialization results in NaNs or Zeros, rendering the model unusable without manual intervention.

Steps to reproduce The following script demonstrates the issue. Note: I used torch.randn for the custom parameter initialization. While I understand that torch.randn samples from a standard normal distribution and does not guarantee an exact sample mean of 0 and std of 1, it should result in valid float values. The observed NaNs/Zeros confirm that this initialization is being discarded and replaced by a faulty internal initialization logic.

Comments:
Repro code throwing "repo_name" errors. Please change the path from "huggingface/Qwen/Qwen3-VL-4B-Instruct" to  "Qwen/Qwen3-VL-4B-Instruct". 

Sorry, i have changed the repo_name

Hi @Noietch, weight initialization for Transformers models is always performed in `post_init()`, which calls `module._init_weights()`. If you want custom weight initialization, then you should override `_init_weights()` in your subclass. For simplicity, you can call `super()._init_weights()` first to initialize most model weights and then just reinitialize the weights you care about.

Hi, @ArthurZucker @Rocketknight1 I’m thinking of adding a skip_post_init flag to from_pretrained(), which would let users opt out of the automatic weight reinitialization step. The default would remain skip_post_init=False, so it wouldn’t affect any existing behavior or break backward compatibility. Would the team be open to a PR implementing this?

I have the same issue loading precomputed weights in a custom model with from_precomputed. loading with torch.load works fine.

Yeah a flag sounds good yes. We have one that we put on the `nn.Parameters` but it completely de-corelated from `torch.load`. 

`torch.load` is fine for very simple use-cases, but really not for any other cases. `from_pretrained` takes care of initializing only what was not loading, missmatched keys, weight materialization scheduling, dtype management, device management, sharding, quantization, etc. 

We should however add a flag it seems to affect a lot of people 🤗 

> the automatic weight reinitialization step

We actually don't intend it to be `automatic` its just that stuff need to be initalized, and instead of copy pasting code from modeling specific, we put it in the `PreTrainedModel`. Unintentionnaly is indeeds affects custom models. 

cc @Cyrilvallez let's change that 🤗 

The tests are failing due to added unexpected argument (flag)😅 😅. 

For now I am updating the migration guide: #42510 

Hey all! Indeed as @ArthurZucker answered, if you want to avoid reinit, you can simply override `_init_weights` in your model, and make it do nothing! You can also check my answer [here](https://github.com/huggingface/transformers/pull/42497#issuecomment-3597054122) -> we may bring that back as default for remote code!

I recently ran into a similar issue, and I was able to resolve it after being inspired by these answers above. However, I am still not entirely sure what it concretely means to “make `_init_weight` do nothing,” because when I override this method and simply use a line of `pass`, I did not observe my issue being resolved.

First, the call chain of my model is roughly as follows:
Module_A → Module_B → Mahalanobis_mask:

```python
class Mahalanobis_mask(nn.Module):
    def __init__(self, input_size):
        super(Mahalanobis_mask, self).__init__()
        frequency_size = input_size // 2
        self.A = nn.Parameter(torch.ones(1, frequency_size), requires_grad=True)
```

Among them, `Mahalanobis_mask` is not constructed during pretraining; it is only additionally initialized when I load the model parameters and perform some custom modifications to the model configurations. As you can see, I need to configure an `nn.Parameter` for it, which is very similar to the situation described in this issue.

After the model finishes loading parameters, I print the weights and find that what is shown is merely a Parameter marked with '*meta*', without any actual numerical values. In any case, this parameter only shows concrete tensor values after formally entering the `forward` process. At that point, I re-print it and observe that `self.A` contains a mixture of many `0.0e+00`s and extremely large or small numbers such as `1.71e+41`, which eventually causes the model to crash as soon as training begins.

As for the solution I eventually found, well it still requires rewriting `_init_weight` at the outset. However, I add a function in `Mahalanobis_mask` with a `reset_parameters` method (or any parameter-initialization function under a different name, whatever~). I simply forced it to allocate and assign values. This does not require much additional codes. Therefore, the modified code is as follows:
```python
# This is the root class A I mentioned
class Transformer_Based_ForPretrain(Transformer_Based_Model):
    def __init__(self, config):
        super().__init__(config)
        self.config = config
        ......
        self.post_init()
    
    def _init_weights(self, module):
        if hasattr(module, "reset_parameters"):
            module.reset_parameters()
        # Those default modules in pytorch also contain reset_parameters, customly reset them again later
        if isinstance(module, torch.nn.Linear):
        ......

......
class Mahalanobis_mask(nn.Module):
    def __init__(self, input_size):
        super(Mahalanobis_mask, self).__init__()
        frequency_size = input_size // 2
        self.A = nn.Parameter(torch.ones(1, frequency_size), requires_grad=True)

    def reset_parameters(self):
        nn.init.constant_(self.A, 1.0)
```

I have not yet figure out the exact workflow that `transformers` uses internally to perform parameter initialization, because during pretraining there is no issue with assigning initial values to any `nn.Parameter`s. In the context of this issue, even if an `nn.Parameter` is forcibly overwritten by some built-in mechanism in `transformers` during initialization, it still should not be filled with strange values such as 0.0e+00 or xxe+41? For that reason, I am actually more curious about what is causing this behavior in the first place.

Hey @Leopold2333, `_init_weights` is responsible for all weight initializations. So you should include your `nn.Parameter` in there if you want it to be correctly initialized. Indeed, it's otherwise just filled with random values (whatever is in memory at that point, according to `torch.empty`)