Add heterogeneous config support (per-layer configuration)
Author: eladsegal
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=45333)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45333)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Adds heterogeneous model support - the ability for individual layers to differ from the global config (e.g., different `intermediate_size`, `num_key_value_heads`) and to skip sub-modules entirely (MLP, attention, etc.). This enables models where layers are not uniform, as in pruned, distilled, or NAS-derived architectures.

Examples of such models:

| Model | Derived from |
|---|---|
| [nvidia/Llama-3_3-Nemotron-Super-49B-v1_5](https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5) | [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) |
| [nvidia/Llama-3_1-Nemotron-Ultra-253B-v1](https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1) | [meta-llama/Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct) |
| [nvidia/gpt-oss-puzzle-88B](https://huggingface.co/nvidia/gpt-oss-puzzle-88B) | [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b/tree/main) |

These models previously required `trust_remote_code=True` to modify the classes of the model they are derived from. With this PR (and the follow-up modeling PR #45332), heterogeneous support requires just a few lines.

This PR contains the configuration layer only. The modeling, cache, and masking changes that consume this config are in #45332.

## How it works

### Configuration (`per_layer_config`)

A new `per_layer_config` parameter on `PreTrainedConfig` maps layer indices to attribute overrides:

```python
from transformers import LlamaConfig

config = LlamaConfig(
    ...,
    per_layer_config={
        0: {"intermediate_size": 64},
        2: {"intermediate_size": 96, "skip": ["attention"]},
    },
)
```

Under the hood, `apply_heterogeneous_config` validates the overrides, computes fallback values, and stores a `HeterogeneitySpec` on the config. Serialization keeps `per_layer_config` as a sparse mapping of layer indices to changed attributes only. At runtime, `config.per_layer_config[i]` returns a full layer config built from the current global config plus that layer's overrides. When a per-layer override matches the global config value, it is omitted from the serialized per_layer_config so the sparse config only records real differences. The config supports full `save_pretrained` / `from_pretrained` round-trips (keys are zero-padded for correct JSON sort order).

Accessing a per-layer attribute on the global config raises `AttributeError` and points callers to `config.per_layer_config[i]`, since the global value can be misleading. Callers can set `allow_global_per_layer_attribute_access=True` to allow `config.<per_layer_attribute>` access, but it will output a `warning_once` message.

## Key changes

- **New `src/transformers/heterogeneity/` package** - configuration utilities (`HeterogeneitySpec`, validation, runtime per-layer config view, serialization)
- **`configuration_utils.py`** - `per_layer_config` property/view, `is_heterogeneous`, serialization hooks, and `__getattribute__` guard for per-layer attributes

## Tests

Test suite in `tests/heterogeneity/test_configuration_utils.py` covering per-layer overrides and fallback, skip normalization, validation, global per-layer attribute access, and save/load round-trip.

## Who can review?
@ArthurZucker
@hmellor

Comments:
@askliar
Related to https://github.com/vllm-project/vllm/pull/36512

Thanks @ArthurZucker! I agree with the explicit and simple direction.

The code now keeps serialization minimal, and exposes full per-layer configs:
- On disk / in `to_dict()`, `per_layer_config` is a sparse mapping of layer index -> changed attributes only, e.g. `{"1": {"hidden_size": 2048}}`.
- At runtime, `config.per_layer_config[i]` returns a full `PreTrainedConfig` built from the *current* base config plus that layer's overrides. Using the current base config matters because model initialization can mutate config state, for example `_attn_implementation_internal`.
- Values matching the global config are omitted, empty layer entries are dropped, and direct global access to varying per-layer attrs raises `AttributeError` unless explicitly opted into.

So we keep the config file simple and minimal for both input and serialization, while the instantiated config object provides convenient and intuitive per-layer access.

I still have a few concerns about injection of heterogenous checks etc but its not blocking anymore IMO

Thanks for the review @ArthurZucker!
I believe I addressed all of the issues raised in your comments.

Regarding disabling the function of the inherited class / enable based on `is_heterogeneous` - If I understand correctly, that's exactly the case. All of the functionality provided by `HeterogeneousConfigMixin` is enabled only if the config is indeed heterogeneous, and otherwise there's no effect other than the visibility of the methods and properties added by the mixin.

Overwriting the inherited methods entirely to keep `PreTrainedConfig` clean from changes isn't possible though: the mixin is a base of `PreTrainedConfig`, so `PreTrainedConfig`'s own implementations win the MRO. I think the only way it would have been possible is by deconstructing `PreTrainedConfig` into smaller mixins that will be after `HeterogeneousConfigMixin` in the MRO.
However, where the MRO does allow it, we already rely on it: the per-layer attribute guard is called through `super().__getattribute__`, so `PreTrainedConfig.__getattribute__` is unchanged compared to main.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45333). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45333)
**Latest run:** [29015415548:2](https://github.com/huggingface/transformers/actions/runs/29027002260)
**Result:** `success` | **Jobs:** 15 | **Tests:** 170,757 | **Failures:** 1 | **Duration:** 4h 3m

<!-- ci-dashboard-recap:end -->