Fix legacy Qwen3-ASR checkpoint loading
Author: Sunt-ing
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47063)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47063)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

The original Qwen3-ASR checkpoints still use the pre-conversion layout:

- `Qwen/Qwen3-ASR-0.6B`
- `Qwen/Qwen3-ASR-1.7B`
- `Qwen/Qwen3-ForcedAligner-0.6B`

Their config stores the native audio/text configs under `thinker_config`, and their weights use `thinker.*` names. Current native loading ignores that nested config, so `AutoConfig.from_pretrained("Qwen/Qwen3-ASR-0.6B")` builds the default Qwen3-ASR config instead of the 0.6B config. Loading a legacy state dict then leaves all `thinker.*` weights unexpected and initializes the native weights as missing.

This PR adds the same compatibility that the offline Qwen3-ASR converter already applies:

- unfold legacy `thinker_config` only when native `audio_config` / `text_config` are absent
- normalize the old audio config name `qwen3_asr_audio_encoder`
- convert legacy `rope_scaling` / `rope_theta` into native `rope_parameters`
- use `classify_num` as `num_labels` for the forced aligner
- map legacy `thinker.*` checkpoint keys to the native ASR and forced-aligner model names at load time
- map ASR `thinker.lm_head.*` to `lm_head.*` and forced-aligner `thinker.lm_head.*` to `score.*`

The already converted `-hf` checkpoints keep the existing path.

On the public original checkpoints, current main (`b70d02fc72`) leaves `Qwen/Qwen3-ASR-0.6B` with 708 missing / 612 unexpected keys and `Qwen/Qwen3-ForcedAligner-0.6B` with 708 missing / 708 unexpected keys. This branch loads both with 0 missing / 0 unexpected keys.

The regression test writes a small legacy-format checkpoint to disk and loads it through `from_pretrained`, so CI covers the same runtime loading path without depending on Hub downloads of the public Qwen checkpoints.

<details>
<summary>End-to-end from_pretrained repro (script + full output)</summary>

Run from a Transformers checkout:

```bash
CUDA_VISIBLE_DEVICES="" PYTHONPATH=$PWD/src python repro_qwen3_asr_legacy.py
```

```python
import json
from tempfile import TemporaryDirectory

import torch
from safetensors.torch import save_file

from transformers import (
    AutoConfig,
    Qwen3ASRConfig,
    Qwen3ASREncoderConfig,
    Qwen3ASRForConditionalGeneration,
    Qwen3ASRForTokenClassification,
    Qwen3Config,
)


def show_public_config(repo_id):
    config = AutoConfig.from_pretrained(repo_id)
    print(
        "public_config",
        repo_id,
        config.audio_config.d_model,
        config.audio_config.output_dim,
        config.audio_config.encoder_layers,
        config.text_config.hidden_size,
        getattr(config, "num_labels", None),
    )


def native_config(num_labels=None):
    config = Qwen3ASRConfig(
        audio_config=Qwen3ASREncoderConfig(
            num_mel_bins=20,
            d_model=16,
            encoder_layers=1,
            encoder_attention_heads=2,
            encoder_ffn_dim=16,
            output_dim=16,
            downsample_hidden_size=4,
            max_position_embeddings=13,
        ),
        text_config=Qwen3Config(
            vocab_size=99,
            hidden_size=16,
            intermediate_size=16,
            num_hidden_layers=1,
            num_attention_heads=2,
            num_key_value_heads=2,
            head_dim=8,
            max_position_embeddings=64,
            tie_word_embeddings=True,
        ),
        audio_token_id=98,
        pad_token_id=0,
        eos_token_id=[1, 2],
    )
    if num_labels is not None:
        config.num_labels = num_labels
    return config


def to_legacy_key(key):
    if key.startswith("model.multi_modal_projector.linear_1"):
        return "thinker.audio_tower.proj1" + key[len("model.multi_modal_projector.linear_1") :]
    if key.startswith("model.multi_modal_projector.linear_2"):
        return "thinker.audio_tower.proj2" + key[len("model.multi_modal_projector.linear_2") :]
    if key.startswith("model.language_model"):
        return "thinker.model" + key[len("model.language_model") :]
    if key.startswith("model.audio_tower"):
        return "thinker.audio_tower" + key[len("model.audio_tower") :]
    if key.startswith("lm_head"):
        return "thinker.lm_head" + key[len("lm_head") :]
    if key.startswith("score"):
        return "thinker.lm_head" + key[len("score") :]
    return key


def run_legacy_checkpoint(model_class, num_labels, head_key):
    config = native_config(num_labels)
    model = model_class(config).eval()
    legacy_state = {to_legacy_key(key): value.detach().clone() for key, value in model.state_dict().items()}

    config_dict = config.to_dict()
    legacy_config = {
        "architectures": [model_class.__name__],
        "model_type": "qwen3_asr",
        "thinker_config": {
            "audio_config": config_dict["audio_config"],
            "text_config": config_dict["text_config"],
            "audio_token_id": config.audio_token_id,
            "pad_token_id": config.pad_token_id,
            "eos_token_id": config.eos_token_id,
            "tie_word_embeddings": config.tie_word_embeddings,
            "token_classification_bias": config.token_classification_bias,
        },
    }
    if num_labels is not None:
        legacy_config["thinker_config"]["classify_num"] = num_labels
    legacy_config["thinker_config"]["audio_config"]["model_type"] = "qwen3_asr_audio_encoder"
    legacy_config["thinker_config"]["text_config"]["rope_scaling"] = {"type": "default", "rope_type": "default"}
    legacy_config["thinker_config"]["text_config"].pop("rope_parameters", None)
    legacy_config["thinker_config"]["text_config"]["rope_theta"] = 1000000

    with TemporaryDirectory() as tmp_dir:
        with open(f"{tmp_dir}/config.json", "w", encoding="utf-8") as f:
            json.dump(legacy_config, f)
        save_file(legacy_state, f"{tmp_dir}/model.safetensors")

        loaded, info = model_class.from_pretrained(tmp_dir, output_loading_info=True)
        print("load", model_class.__name__, "missing", len(info["missing_keys"]), "unexpected", len(info["unexpected_keys"]))
        loaded_state = loaded.state_dict()
        for old_key, new_key in [
            ("thinker.audio_tower.proj1.weight", "model.multi_modal_projector.linear_1.weight"),
            ("thinker.model.embed_tokens.weight", "model.language_model.embed_tokens.weight"),
            ("thinker.lm_head.weight", head_key),
        ]:
            print("mapped", old_key, "->", new_key, torch.equal(legacy_state[old_key], loaded_state.get(new_key, torch.empty(0))))


if __name__ == "__main__":
    show_public_config("Qwen/Qwen3-ASR-0.6B")
    show_public_config("Qwen/Qwen3-ForcedAligner-0.6B")
    run_legacy_checkpoint(Qwen3ASRForConditionalGeneration, None, "lm_head.weight")
    run_legacy_checkpoint(Qwen3ASRForTokenClassification, 5, "score.weight")
```

Before:

```text
public_config Qwen/Qwen3-ASR-0.6B 1024 3584 24 2048 2
public_config Qwen/Qwen3-ForcedAligner-0.6B 1024 3584 24 2048 2
load Qwen3ASRForConditionalGeneration missing 708 unexpected 43
mapped thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight False
mapped thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight False
mapped thinker.lm_head.weight -> lm_head.weight False
load Qwen3ASRForTokenClassification missing 708 unexpected 43
mapped thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight False
mapped thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight False
mapped thinker.lm_head.weight -> score.weight False
```

After:

```text
public_config Qwen/Qwen3-ASR-0.6B 896 1024 18 1024 2
public_config Qwen/Qwen3-ForcedAligner-0.6B 1024 1024 24 1024 5000
load Qwen3ASRForConditionalGeneration missing 0 unexpected 0
mapped thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight True
mapped thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight True
mapped thinker.lm_head.weight -> lm_head.weight True
load Qwen3ASRForTokenClassification missing 0 unexpected 0
mapped thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight True
mapped thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight True
mapped thinker.lm_head.weight -> score.weight True
```

</details>

<details>
<summary>Validation</summary>

Real public checkpoint load through `from_pretrained(..., output_loading_info=True)`:

```bash
CUDA_VISIBLE_DEVICES="" PYTHONPATH=$PWD/src python - <<'PY'
from transformers import Qwen3ASRForConditionalGeneration, Qwen3ASRForTokenClassification

for repo_id, model_class in [
    ("Qwen/Qwen3-ASR-0.6B", Qwen3ASRForConditionalGeneration),
    ("Qwen/Qwen3-ForcedAligner-0.6B", Qwen3ASRForTokenClassification),
]:
    _, info = model_class.from_pretrained(repo_id, output_loading_info=True)
    print(repo_id, "missing", len(info["missing_keys"]), "unexpected", len(info["unexpected_keys"]))
PY
```

```text
current main b70d02fc72:
Qwen/Qwen3-ASR-0.6B missing 708 unexpected 612
Qwen/Qwen3-ForcedAligner-0.6B missing 708 unexpected 708

patched 905200bae2:
Qwen/Qwen3-ASR-0.6B missing 0 unexpected 0
Qwen/Qwen3-ForcedAligner-0.6B missing 0 unexpected 0
```

Synthetic legacy checkpoint guard:

```text
current main:
Qwen3ASRForConditionalGeneration missing 708 unexpected 43
thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight False
thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight False
thinker.lm_head.weight -> lm_head.weight False
Qwen3ASRForTokenClassification missing 708 unexpected 43
thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight False
thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight False
thinker.lm_head.weight -> score.weight False

patched:
Qwen3ASRForConditionalGeneration missing 0 unexpected 0
thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight True
thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight True
thinker.lm_head.weight -> lm_head.weight True
Qwen3ASRForTokenClassification missing 0 unexpected 0
thinker.audio_tower.proj1.weight -> model.multi_modal_projector.linear_1.weight True
thinker.model.embed_tokens.weight -> model.language_model.embed_tokens.weight True
thinker.lm_head.weight -> score.weight True
```

```text
CUDA_VISIBLE_DEVICES= PYTHONPATH=$PWD/src python -m pytest -q \
  tests/models/qwen3_asr/test_modeling_qwen3_asr.py::Qwen3ASRForConditionalGenerationModelTest::test_legacy_thinker_config \
  tests/models/qwen3_asr/test_modeling_qwen3_asr.py::Qwen3ASRForConditionalGenerationModelTest::test_legacy_thinker_checkpoint_conversion \
  tests/models/qwen3_asr/test_modeling_qwen3_asr.py::Qwen3ASRForConditionalGenerationModelTest::test_can_use_safetensors \
  tests/models/qwen3_asr/test_modeling_qwen3_asr.py::Qwen3ASRForConditionalGenerationModelTest::test_forward_with_logits_to_keep

4 passed

CUDA_VISIBLE_DEVICES= PYTHONPATH=$PWD/src python -m pytest -q \
  tests/models/qwen3_asr/test_modeling_qwen3_asr.py::Qwen3ASRForConditionalGenerationModelTest \
  -k "not torch_export"

137 passed, 110 skipped, 12 deselected

ruff check ... -> All checks passed
ruff format --check ... -> 4 files already formatted
python utils/check_modular_conversion.py --files src/transformers/models/qwen3_asr/modular_qwen3_asr.py -> passed
git diff --check -> passed
```

</details>

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://huggingface.co/docs/transformers/contributing) and the
      [Pull Request](https://huggingface.co/docs/transformers/pr_checks) checks?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes according to the [guidelines](https://github.com/huggingface/transformers/tree/main/docs)?
- [x] Did you write any new necessary [tests](https://huggingface.co/docs/transformers/testing)?

## Who can review?

cc @ebezzam @vasqu

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_asr

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47063)
**Latest run:** [28702132496:2](https://github.com/huggingface/transformers/actions/runs/28702132496)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,466 | **Failures:** 7 | **Duration:** 24h 42m

<!-- ci-dashboard-recap:end -->

Hey @Sunt-ing appreciate the effort but I don't think we actually want to support the legacy checkpoints. Users should use the -hf checkpoints for the usage with transformers. 

Thanks @vasqu, understood.