transformers 5.x: from_pretrained silently returns random-init weights for custom PreTrainedModel subclasses (clean loading_info, no warning)
Author: gkriegspeedbay
Pull request: no

### System Info

- transformers: 5.12.0
- torch: 2.12.0+cpu
- Python: 3.11
- Reproduces in a clean venv with only `transformers`, `torch`, `safetensors` installed (OS-independent).

### Description

A custom (out-of-tree) `PreTrainedModel` subclass saved with `save_pretrained()` and reloaded with `from_pretrained()` comes back with **randomly-initialized weights** instead of the saved values, while `from_pretrained` reports a **clean load**: `missing_keys=[]`, `unexpected_keys=[]`, `mismatched_keys=[]`, `error_msgs=[]`, and no warning.

The saved `model.safetensors` contains the correct weights (verified in the repro), so saving is fine — it's `from_pretrained`'s weight materialization that is silently skipped for the custom class. In-tree architectures (e.g. `DistilBertForSequenceClassification`) load the same checkpoint correctly, so this is specific to custom subclasses.

The silence is the dangerous part: the model looks loaded but is effectively random, with nothing (error, warning, or loading_info) to catch it. I hit this porting a downstream library to transformers 5.x — it surfaced as a model that ran but produced garbage while `from_pretrained` reported success.

### Reproduction

```python
import os, tempfile, torch, torch.nn as nn
from transformers import PreTrainedModel, PretrainedConfig
from safetensors.torch import load_file

class ToyConfig(PretrainedConfig):
    model_type = "toy"
    def __init__(self, dim=32, **kw):
        self.dim = dim
        super().__init__(**kw)

class ToyModel(PreTrainedModel):
    config_class = ToyConfig
    base_model_prefix = "toy"
    def __init__(self, config):
        super().__init__(config)
        self.lin = nn.Linear(config.dim, config.dim)
        self.post_init()
    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            module.weight.data.normal_(mean=0.0, std=0.02)
            if module.bias is not None:
                module.bias.data.zero_()
    def forward(self, x):
        return self.lin(x)

m = ToyModel(ToyConfig(dim=32))
with torch.no_grad():
    for p in m.parameters():
        p.fill_(0.5)

d = tempfile.mkdtemp()
m.save_pretrained(d)
print("RAW saved weight mean:", load_file(os.path.join(d, "model.safetensors"))["lin.weight"].mean().item())

m2, info = ToyModel.from_pretrained(d, output_loading_info=True)
print("loaded weight mean   :", m2.lin.weight.mean().item())
print("loading_info         :", info)
```

Output on transformers 5.12.0:

```
RAW saved weight mean: 0.5
loaded weight mean   : 0.0007
loading_info         : {'missing_keys': [], 'unexpected_keys': [], 'mismatched_keys': [], 'error_msgs': []}
```

### Expected behavior

`from_pretrained` should load the saved weights (as on transformers 4.x), or at minimum surface the un-materialized parameters via `missing_keys`/a warning instead of silently returning a randomly-initialized model.

Comments:
I can reproduce this on current `main` (`transformers==5.13.0.dev0`, `torch==2.12.0` CPU) — same output: saved weight mean `0.5`, reloaded mean `~0.0007`, with a clean `loading_info` (`missing_keys`/`unexpected_keys`/`mismatched_keys`/`error_msgs` all empty).

### Root cause

The weights *are* loaded correctly — they're then re-initialized after loading.

1. During loading, `set_param_for_module` materializes `lin.weight`/`lin.bias` from the checkpoint and flags each loaded param `_is_hf_initialized = True` (`core_model_loading.py`, ~L1132). At this point `lin.weight` is `0.5` and `missing_keys` is empty (params were found), which is why `loading_info` looks clean.
2. `_finalize_model_loading` then calls `_initialize_missing_keys` → `initialize_weights()` → `_initialize_weights(module)` (`modeling_utils.py:2492`).
3. In `_initialize_weights`, the **param-level** backstop — skip a module when all its directly-owned params/buffers are already `_is_hf_initialized` (L2502–2512) — is **gated behind `is_remote_code`**. A custom out-of-tree subclass loaded directly (not via `trust_remote_code`) has `is_remote_code() == False`, so it skips the backstop and falls through to `self._init_weights(module)` (L2514), which runs the user's `module.weight.data.normal_(...)` in place and silently overwrites the loaded `0.5` with random values.

Confirmed with an instrumented run:
```
set_param_for_module: 'lin.weight'  value_mean=0.5
_initialize_missing_keys BEFORE: lin.weight mean = 0.5  _is_hf_initialized = True
  _init_weights CALLED on Linear            # re-inits despite the flag
_initialize_missing_keys AFTER:  lin.weight mean = 0.0007
```

Why in-tree models are unaffected: their `_init_weights` use the guarded `transformers.initialization` helpers (and `@guard_torch_init_functions`), which check `_is_hf_initialized` per tensor. A user writing `weight.data.normal_(...)` operates on `.data` (a distinct tensor without the flag), so it isn't protected — which is exactly the scenario the remote-code backstop was added for, but it's currently limited to remote code.

### Proposed fix

Honor the param-level `_is_hf_initialized` backstop for **all** models, i.e. drop the `is_remote_code` gate in `_initialize_weights` so any module whose directly-owned params/buffers are all already initialized is skipped. Verified locally that this restores correct loading (reloaded mean `0.5`) with no other change. I'd add a regression test doing a save→load round-trip on a minimal custom `PreTrainedModel` and asserting the weights match.

One nuance worth your call: this fully fixes the common case (all of a module's params loaded). A module that is *partially* loaded with an unguarded in-place `_init_weights` would still re-init its loaded params; if you'd rather close that too, an alternative is to propagate the param-level flag to modules in `_initialize_missing_keys` before `initialize_weights()` (matching what that method's docstring already describes).

### Not a duplicate

No open PR addresses this (checked). This looks like the same family as #42418; that issue's PR #42497 (`skip_post_init` flag) was closed — this proposes a root-cause fix rather than a user-facing flag.

Happy to open a PR with the fix + regression test if this direction looks right to you — or adjust to whichever approach you prefer.


Thanks for the deep dive — this matches what I saw (weights load, then get re-initialized; `loading_info` looks clean because the params *were* found before `_init_weights` overwrites them), and it explains the in-tree/custom split nicely.

I independently verified the proposed fix on `transformers==5.12.0`: dropping the `is_remote_code` gate on the param-level `_is_hf_initialized` backstop in `_initialize_weights` restores correct loading — round-tripped weight mean goes back to `0.5` (was `~0.0007`), with no other change.

Please do open the PR — happy to test it against the real-world case that surfaced this (a downstream OCR library with custom `PreTrainedModel` subclasses; I have both a minimal save→load harness and an end-to-end check) and to review.

I'd like a little more explanation here - can someone do a bisect to find the commit where this bug was introduced?

Bisected it — **first bad commit is `6f6095e0cf509f7384d3ce0c1804013ef6cafd5f`, "Refactor weight loading" (#41580)**.

Ran the custom-subclass save→load round-trip at each step over `c4cfc2e02..main` (good = the commit right before #41580, bad = current main):

- parent `c4cfc2e02` ([TP] fix #42129): reloaded weight mean **0.5** ✅
- `6f6095e0cf` (#41580): reloaded weight mean **~0.001** ❌ (silently re-initialized)

They're adjacent on main, so #41580 is the regression point. It's the PR that added `core_model_loading.py` and the new materialization path. At that commit `_initialize_weights` only honored the *module-level* `_is_hf_initialized` flag:

```python
def _initialize_weights(self, module):
    if getattr(module, "_is_hf_initialized", False):
        return
    self._init_weights(module)
```

The new loader sets `_is_hf_initialized` per-*param*, not on the module, so for a custom subclass the module flag is False, `_init_weights(module)` runs, and the just-loaded weights get overwritten. No param-level backstop existed at this commit.

The param-level backstop referenced in the root-cause analysis was added later, in **#44095 "Fix loading logic issue"** (`a64996e33b`) — but scoped behind `is_remote_code`, so it only ever covered remote-code models and left in-tree-loaded custom subclasses (`is_remote_code() == False`) regressing as introduced by #41580.

Dropping the `is_remote_code` gate on that backstop (the proposed fix) restores correct loading on current `main` (`5.13.0.dev0`): reloaded mean 0.0007 → 0.5, no other change.

(Repro built each commit from source, torch 2.12.0+cpu, py 3.11. Minor note for anyone reproducing on Windows: #41580 also has an incidental `checkpoint_files[0].rsplit("/", 1)[1]` path bug that raises IndexError on backslash paths — fixed before v5.0.0; use a POSIX-style temp path and the silent re-init shows through.)


A lot of issues and misunderstanding already around this. This is because of your `_init_weights`. Please use either `torch.init` module or `transformer.initialization` module to init modules inside `_init_weights`, instead of updating them in-place. This will fix the issue
Let me know if something is still unclear!