Can not deploy SFT Qwen3.5-9B model
Author: zouYC2021
Pull request: no
Labels: name: bug

### System Info

I SFT Qwen3.5-9B model with transformers==5.2.0

### Reproduction

When I try to deploy my model with vllm==0.17.0, it reports:

TypeError: Invalid type of HuggingFace config. Expected type: <class 'vllm.transformers_utils.configs.qwen3_5.Qwen3_5Config'>, but found type: <class 'transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig'>

I wonder how can I fix it.

### Expected behavior

vllm==0.17.0 requires transformers<5, but transformers>5 is required to support Qwen3.5, kind of strange.

Comments:
I'm not sure we can fix bugs on older versions of Transformers - have you tried the latest version?

> I'm not sure we can fix bugs on older versions of Transformers - have you tried the latest version?

Yep, I got same error with transformers==5.3.0

cc @zucchini-nlp maybe, or check with Harry?

vLLM vendored everything for Qwen 3.5 because they couldn't wait for Transformers v5 support.

So there's a mismatch with the vendored config and the upstream config.

@hmellor Is there an existing solution or suggested workaround for this issue?

I don't actually see this error (after fixing a couple of things which changed in the last day). 

Can you try https://github.com/vllm-project/vllm/pull/37398? I am able to serve Qwen 3.5 with it.

This is an issue with the SFT tool or the quantization tool; you can share the model's config file, which should differ from the original model.

config.json in origin Qwen3.5-9B :
```
{
    "architectures": [
        "Qwen3_5ForConditionalGeneration"
    ],
    "image_token_id": 248056,
    "model_type": "qwen3_5",
    "text_config": {
        "attention_bias": false,
        "attention_dropout": 0.0,
        "attn_output_gate": true,
        "dtype": "bfloat16",
        "eos_token_id": 248044,
        "full_attention_interval": 4,
        "head_dim": 256,
        "hidden_act": "silu",
        "hidden_size": 4096,
        "initializer_range": 0.02,
        "intermediate_size": 12288,
        "layer_types": [
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention"
        ],
        "linear_conv_kernel_dim": 4,
        "linear_key_head_dim": 128,
        "linear_num_key_heads": 16,
        "linear_num_value_heads": 32,
        "linear_value_head_dim": 128,
        "max_position_embeddings": 262144,
        "mlp_only_layers": [],
        "model_type": "qwen3_5_text",
        "mtp_num_hidden_layers": 1,
        "mtp_use_dedicated_embeddings": false,
        "num_attention_heads": 16,
        "num_hidden_layers": 32,
        "num_key_value_heads": 4,
        "rms_norm_eps": 1e-06,
        "use_cache": true,
        "vocab_size": 248320,
        "mamba_ssm_dtype": "float32",
        "rope_parameters": {
            "mrope_interleaved": true,
            "mrope_section": [
                11,
                11,
                10
            ],
            "rope_type": "default",
            "rope_theta": 10000000,
            "partial_rotary_factor": 0.25
        }
    },
    "tie_word_embeddings": false,
    "transformers_version": "4.57.0.dev0",
    "video_token_id": 248057,
    "vision_config": {
        "deepstack_visual_indexes": [],
        "depth": 27,
        "hidden_act": "gelu_pytorch_tanh",
        "hidden_size": 1152,
        "in_channels": 3,
        "initializer_range": 0.02,
        "intermediate_size": 4304,
        "model_type": "qwen3_5",
        "num_heads": 16,
        "num_position_embeddings": 2304,
        "out_hidden_size": 4096,
        "patch_size": 16,
        "spatial_merge_size": 2,
        "temporal_patch_size": 2
    },
    "vision_end_token_id": 248054,
    "vision_start_token_id": 248053
}
```

config.json after SFT:
```
{
  "architectures": [
    "Qwen3_5ForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "attn_output_gate": true,
  "bos_token_id": null,
  "dtype": "bfloat16",
  "eos_token_id": 248046,
  "full_attention_interval": 4,
  "head_dim": 256,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 12288,
  "layer_types": [
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention"
  ],
  "linear_conv_kernel_dim": 4,
  "linear_key_head_dim": 128,
  "linear_num_key_heads": 16,
  "linear_num_value_heads": 32,
  "linear_value_head_dim": 128,
  "mamba_ssm_dtype": "float32",
  "max_position_embeddings": 262144,
  "mlp_only_layers": [],
  "model_type": "qwen3_5_text",
  "mtp_num_hidden_layers": 1,
  "mtp_use_dedicated_embeddings": false,
  "num_attention_heads": 16,
  "num_hidden_layers": 32,
  "num_key_value_heads": 4,
  "pad_token_id": 248044,
  "partial_rotary_factor": 0.25,
  "rms_norm_eps": 1e-06,
  "rope_parameters": {
    "mrope_interleaved": true,
    "mrope_section": [
      11,
      11,
      10
    ],
    "partial_rotary_factor": 0.25,
    "rope_theta": 10000000,
    "rope_type": "default"
  },
  "tie_word_embeddings": false,
  "transformers_version": "5.2.0",
  "use_cache": false,
  "vocab_size": 248320
}
```

> config.json in origin Qwen3.5-9B :
> 
> ```
> {
>     "architectures": [
>         "Qwen3_5ForConditionalGeneration"
>     ],
>     "image_token_id": 248056,
>     "model_type": "qwen3_5",
>     "text_config": {
>         "attention_bias": false,
>         "attention_dropout": 0.0,
>         "attn_output_gate": true,
>         "dtype": "bfloat16",
>         "eos_token_id": 248044,
>         "full_attention_interval": 4,
>         "head_dim": 256,
>         "hidden_act": "silu",
>         "hidden_size": 4096,
>         "initializer_range": 0.02,
>         "intermediate_size": 12288,
>         "layer_types": [
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention"
>         ],
>         "linear_conv_kernel_dim": 4,
>         "linear_key_head_dim": 128,
>         "linear_num_key_heads": 16,
>         "linear_num_value_heads": 32,
>         "linear_value_head_dim": 128,
>         "max_position_embeddings": 262144,
>         "mlp_only_layers": [],
>         "model_type": "qwen3_5_text",
>         "mtp_num_hidden_layers": 1,
>         "mtp_use_dedicated_embeddings": false,
>         "num_attention_heads": 16,
>         "num_hidden_layers": 32,
>         "num_key_value_heads": 4,
>         "rms_norm_eps": 1e-06,
>         "use_cache": true,
>         "vocab_size": 248320,
>         "mamba_ssm_dtype": "float32",
>         "rope_parameters": {
>             "mrope_interleaved": true,
>             "mrope_section": [
>                 11,
>                 11,
>                 10
>             ],
>             "rope_type": "default",
>             "rope_theta": 10000000,
>             "partial_rotary_factor": 0.25
>         }
>     },
>     "tie_word_embeddings": false,
>     "transformers_version": "4.57.0.dev0",
>     "video_token_id": 248057,
>     "vision_config": {
>         "deepstack_visual_indexes": [],
>         "depth": 27,
>         "hidden_act": "gelu_pytorch_tanh",
>         "hidden_size": 1152,
>         "in_channels": 3,
>         "initializer_range": 0.02,
>         "intermediate_size": 4304,
>         "model_type": "qwen3_5",
>         "num_heads": 16,
>         "num_position_embeddings": 2304,
>         "out_hidden_size": 4096,
>         "patch_size": 16,
>         "spatial_merge_size": 2,
>         "temporal_patch_size": 2
>     },
>     "vision_end_token_id": 248054,
>     "vision_start_token_id": 248053
> }
> ```
> 
> config.json after SFT:
> 
> ```
> {
>   "architectures": [
>     "Qwen3_5ForCausalLM"
>   ],
>   "attention_bias": false,
>   "attention_dropout": 0.0,
>   "attn_output_gate": true,
>   "bos_token_id": null,
>   "dtype": "bfloat16",
>   "eos_token_id": 248046,
>   "full_attention_interval": 4,
>   "head_dim": 256,
>   "hidden_act": "silu",
>   "hidden_size": 4096,
>   "initializer_range": 0.02,
>   "intermediate_size": 12288,
>   "layer_types": [
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention"
>   ],
>   "linear_conv_kernel_dim": 4,
>   "linear_key_head_dim": 128,
>   "linear_num_key_heads": 16,
>   "linear_num_value_heads": 32,
>   "linear_value_head_dim": 128,
>   "mamba_ssm_dtype": "float32",
>   "max_position_embeddings": 262144,
>   "mlp_only_layers": [],
>   "model_type": "qwen3_5_text",
>   "mtp_num_hidden_layers": 1,
>   "mtp_use_dedicated_embeddings": false,
>   "num_attention_heads": 16,
>   "num_hidden_layers": 32,
>   "num_key_value_heads": 4,
>   "pad_token_id": 248044,
>   "partial_rotary_factor": 0.25,
>   "rms_norm_eps": 1e-06,
>   "rope_parameters": {
>     "mrope_interleaved": true,
>     "mrope_section": [
>       11,
>       11,
>       10
>     ],
>     "partial_rotary_factor": 0.25,
>     "rope_theta": 10000000,
>     "rope_type": "default"
>   },
>   "tie_word_embeddings": false,
>   "transformers_version": "5.2.0",
>   "use_cache": false,
>   "vocab_size": 248320
> }
> ```

This is an incorrect configuration file, and it's obvious that multimodal-related data is missing; you can go to the project of the SFT tool to submit an issue.

Hi! @zouYC2021, thank you for this issue.
Could you please share which SFT tool you used and how you ran it?
Thanks!

I use trl==0.29.0 as SFT tool. My train_sft.py is as follows:
```
import os
import argparse
import torch
from pathlib import Path
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig
from transformers.trainer_utils import get_last_checkpoint

def parse_args():
    ap = argparse.ArgumentParser()
    ap.add_argument("--model_dir", required=True, type=str)
    ap.add_argument("--train_file", required=True, type=str)
    ap.add_argument("--val_file", required=True, type=str)
    ap.add_argument("--out_dir", required=True, type=str)
    ap.add_argument("--epochs", type=int, default=2)
    ap.add_argument("--lr", type=float, default=2e-5)
    ap.add_argument("--grad_accum", type=int, default=16)
    ap.add_argument("--max_len", type=int, default=4096)
    ap.add_argument("--seed", type=int, default=42)
    ap.add_argument("--deepspeed_config", type=str, default=None)
    return ap.parse_args()

def collect_jsonl_files(path_str: str):
    p = Path(path_str)
    files = []
    if p.is_dir():
        files = sorted(str(f) for f in p.rglob("*.jsonl"))
    elif p.is_file():
        if p.suffix == ".jsonl":
            files = [str(p)]
        else:
            raise ValueError(f"Expected a .jsonl file, got: {p}")
    else:
        files = sorted(str(f) for f in Path().glob(path_str))

    if not files:
        raise FileNotFoundError(f"No .jsonl files found under: {path_str}")
    return files


def main():
    args = parse_args()
    torch.manual_seed(args.seed)

    # ---- Tokenizer / Model ----
    tok = AutoTokenizer.from_pretrained(
        args.model_dir, use_fast=False, trust_remote_code=True
    )
    if tok.pad_token is None and tok.eos_token is not None:
        tok.pad_token = tok.eos_token

    model = AutoModelForCausalLM.from_pretrained(
        args.model_dir,
        torch_dtype=torch.bfloat16,
        trust_remote_code=True
    )
    model.gradient_checkpointing_enable()
    model.config.use_cache = False
    if tok.pad_token_id is not None:
        model.config.pad_token_id = tok.pad_token_id
    if tok.eos_token_id is not None:
        model.config.eos_token_id = tok.eos_token_id

    data_files = {
        "train": args.train_file,
        "validation": args.val_file,
    }
    raw_datasets = load_dataset("json", data_files=data_files)

    train_ds = raw_datasets["train"]
    eval_ds = raw_datasets["validation"]

    # ---- Qwen3.5 only ----
    train_ds = train_ds.add_column(
        "chat_template_kwargs",
        [{"enable_thinking": False} for _ in range(len(train_ds))]
    )
    eval_ds = eval_ds.add_column(
        "chat_template_kwargs",
        [{"enable_thinking": False} for _ in range(len(eval_ds))]
    )

    cfg = SFTConfig(
        output_dir=args.out_dir,
        deepspeed=args.deepspeed_config,     # ZeRO-3
        completion_only_loss=True,
        assistant_only_loss=False,

        max_length=args.max_len,
        packing=False,
        eval_packing=False,

        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=args.grad_accum,
        num_train_epochs=args.epochs,
        bf16=True,
        gradient_checkpointing=True,

        learning_rate=args.lr,
        lr_scheduler_type="cosine",
        warmup_ratio=0.03,
        weight_decay=0.1,
        max_grad_norm=1.0,

        save_strategy="steps", save_steps=100,
        eval_strategy="steps", eval_steps=100,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,

        logging_steps=100,
        dataset_num_proc=max(1, (os.cpu_count() or 4) // 2),
        seed=args.seed,

        report_to=["swanlab"],
        run_name="xxx",
    )

    trainer = SFTTrainer(
        model=model,
        args=cfg,
        train_dataset=train_ds,
        eval_dataset=eval_ds,
        processing_class=tok,
    )

    print("[INFO] Start SFT training ...")
    resume_from_checkpoint = False
    if not resume_from_checkpoint:
        trainer.train()
    else:
        print("loading ckpt...")
        trainer.train(resume_from_checkpoint=get_last_checkpoint(args.model_dir))
    trainer.save_model(args.out_dir)
    tok.save_pretrained(args.out_dir)


if __name__ == "__main__":
    main()
```

@zouYC2021 
Came across this bug today, after training a Qwen3.5 model with `trl`.

### Quick and dirty fix:

Replace new `config.json` produced after training with the one before that:

```bash
mv $TRL_CHECKPOINT_PATH/config.json $TRL_CHECKPOINT_PATH/config.json.old # backup 
cp $ORIGINAL_CHECKPOINT_PATH/config.json $TRL_CHECKPOINT_PATH/config.json

```

Same for `tokenizer_config.json`.

### Wait for the bug to be really fixed in future `vllm` release

You are doing SFT on a model loaded with `AutoModelForCausalLM`, which will naturally result in a text only model.

If you want to preserve the config structure since this is a multimodal model you should use `AutoModelForMultimodalLM`.

**Got same issue** 
simplejson                               3.20.2
six                                      1.17.0
sniffio                                  1.3.1
sortedcontainers                         2.4.0
sse-starlette                            3.3.4
starlette                                0.52.1
supervisor                               4.3.0
sympy                                    1.14.0
tabulate                                 0.10.0
tensorboard                              2.20.0
tensorboard-data-server                  0.7.2
tiktoken                                 0.12.0
tokenizers                               0.22.2
tomlkit                                  0.13.3
torch                                    2.10.0+cu128
torch-c-dlpack-ext                       0.1.5
torchaudio                               2.10.0+cu128
torchvision                              0.25.0+cu128
tqdm                                     4.67.3
transformers                             5.3.0
transformers-stream-generator            0.0.5
triton                                   3.6.0
trl                                      0.29.1
typer                                    0.24.1
typer-slim                               0.24.0
typing-extensions                        4.15.0
typing-inspection                        0.4.2
tzdata                                   2026.1
urllib3                                  2.6.3
uvicorn                                  0.44.0
uvloop                                   0.22.1
vllm                                     0.19.0

> You are doing SFT on a model loaded with `AutoModelForCausalLM`, which will naturally result in a text only model.
> 
> If you want to preserve the config structure since this is a multimodal model you should use `AutoModelForMultimodalLM`.

Thanks, you are right. I successfully trained Qwen3.5-9B and deployed SFT model with two modification:

1. Load Qwen3.5-9B with AutoModelForMultimodalLM in train.py. The config.json would remain unchanged before and after SFT.
```
model = AutoModelForMultimodalLM.from_pretrained(
    args.model_dir,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)
```

2. Deploy SFT model with text-only config
```
CUDA_VISIBLE_DEVICES=${GPU_ID} python -m vllm.entrypoints.openai.api_server \
        --model ${SFT_MODEL_PATH} \
        --tensor-parallel-size 1 \
        --port ${PORT} \
        --max_model_len 10000 \
        --enable_prefix_caching \
        --trust-remote-code
        --language-model-only # important
```

use my code to transfer original class name to new name, where vllm0.18 version supported now
`#!/usr/bin/env python3
"""
将 Qwen3.5 CausalLM (纯文本SFT) 模型转换为 ConditionalGeneration (多模态) 格式
只改权重前缀和 config.json，不改模型能力

转换内容:
  - 权重 key: "model.xxx" → "language_model.model.xxx"
  - 权重 key: "lm_head.xxx" → "language_model.lm_head.xxx"
  - config.json: model_type "qwen3_5_text" → "qwen3_5"
  - config.json: architectures "Qwen3_5ForCausalLM" → "Qwen3_5ForConditionalGeneration"

转换后可直接用 vLLM 0.18 原生部署（无需 patch）:
  vllm serve <model_path> --language-model-only --trust-remote-code
"""

import json
import shutil
import sys
from pathlib import Path

from safetensors.torch import load_file, save_file


def convert_model(src_dir, dst_dir=None):
    src_dir = Path(src_dir)
    if dst_dir is None:
        dst_dir = src_dir.parent / (src_dir.name + "-vl-format")
    dst_dir = Path(dst_dir)

    if dst_dir.exists():
        print(f"[ERROR] 目标目录已存在: {dst_dir}")
        sys.exit(1)

    dst_dir.mkdir(parents=True)
    print(f"[INFO] 源目录: {src_dir}")
    print(f"[INFO] 目标目录: {dst_dir}")

    # 1. 转换 safetensors 权重
    safetensor_files = list(src_dir.glob("*.safetensors"))
    if not safetensor_files:
        print("[ERROR] 未找到 .safetensors 文件")
        sys.exit(1)

    for sf in safetensor_files:
        if sf.name.endswith(".bak"):
            continue
        print(f"\n[CONVERT] {sf.name}")
        tensors = load_file(str(sf))

        new_tensors = {}
        for key, value in tensors.items():
            if key.startswith("model."):
                new_key = "language_model." + key
            elif key.startswith("lm_head."):
                new_key = "language_model." + key
            else:
                new_key = key
            new_tensors[new_key] = value
            if new_key != key:
                print(f"  {key} → {new_key}")

        save_file(new_tensors, str(dst_dir / sf.name))
        print(f"  saved: {dst_dir / sf.name}")

    # 2. 转换 config.json
    print(f"\n[CONVERT] config.json")
    config_path = src_dir / "config.json"
    with open(config_path) as f:
        config = json.load(f)

    config["model_type"] = "qwen3_5"
    config["architectures"] = ["Qwen3_5ForConditionalGeneration"]
    print(f"  model_type: qwen3_5_text → qwen3_5")
    print(f"  architectures: Qwen3_5ForCausalLM → Qwen3_5ForConditionalGeneration")

    with open(dst_dir / "config.json", "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)

    # 3. 复制其他文件
    for f in src_dir.iterdir():
        if f.name.endswith(".safetensors") or f.name == "config.json" or f.name.endswith(".bak"):
            continue
        print(f"[COPY] {f.name}")
        if f.is_file():
            shutil.copy2(f, dst_dir / f.name)

    print(f"\n[DONE] 转换完成!")
    print(f"  目标目录: {dst_dir}")
    print(f"\n  部署命令:")
    print(f"  vllm serve {dst_dir} --language-model-only --trust-remote-code")


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print(f"用法: python3 {sys.argv[0]} <源模型目录> [目标目录]")
        print(f"示例: python3 {sys.argv[0]} ./qwen35-sft-merged ./qwen35-sft-merged-vl")
        sys.exit(1)

    src = sys.argv[1]
    dst = sys.argv[2] if len(sys.argv) > 2 else None
    convert_model(src, dst)
`

I tried the code, thanks, itt gets past the error but now  get a new one: OSError: Can't load image processor for '/home/user/Desktop/qwen25base_4bit_vllm'. If you were trying to load it from


@zouYC2021 @hmellor does this mean that even if I'm training on text only, we need to load the model with `AutoModelForMultimodalLM`? If this is the case, shouldn't the library enforce that multimodal models need to be loaded with `AutoModelForMultimodalLM` to prevent these types of issues?

> does this mean that even if I'm training on text only, we need to load the model with AutoModelForMultimodalLM? 

You could use `AutoModelForCausalLM` but you would then need to transplant the text model into a `Qwen3_5ForConditionalGeneration` before saving.

> If this is the case, shouldn't the library enforce that multimodal models need to be loaded with AutoModelForMultimodalLM to prevent these types of issues?

It is perfectly valid to load a multimodal model using `AutoModelForCausalLM` if you don't intend to use the multi-model part.

> > does this mean that even if I'm training on text only, we need to load the model with AutoModelForMultimodalLM?
> 
> You could use `AutoModelForCausalLM` but you would then need to transplant the text model into a `Qwen3_5ForConditionalGeneration` before saving.
> 
> > If this is the case, shouldn't the library enforce that multimodal models need to be loaded with AutoModelForMultimodalLM to prevent these types of issues?
> 
> It is perfectly valid to load a multimodal model using `AutoModelForCausalLM` if you don't intend to use the multi-model part.

@hmellor Thanks for the reply! Just to be clear, by "transplant", you mean doing what @LuisVasquezBSC suggested, right?

```
Replace new config.json produced after training with the one before that:
```

I meant copying the weights from the `ForCausalLM` model into the `text_model` of the `ForMultimodalLM`

@hmellor So do you mean that if I use `AutoModelForCausalLM`, after training, I must edit the `.safetensor` files so that they contain the vision weights from the base model (obtained from the base `.safetensor` files on [HF](https://huggingface.co/Qwen/Qwen3.5-27B/blob/main/model.safetensors-00011-of-00011.safetensors))? After this transplant, I can deploy to vLLM as normally. This seems a bit tedious. 

As an alternative, I went with this approach `Replace new config.json produced after training with the one before that:
` and it worked quite well. I just needed to replace the `config.json` after training with the base model's `config.json` (https://huggingface.co/Qwen/Qwen3.5-27B/blob/main/config.json). 

I imagine both these approaches are equivalent, but please let me know if not. Thanks!

```py
# You trained this:
llm: Qwen3_5ForCausalLM
# But vLLM wants to load this:
vlm: Qwen3_5ForConditionalGeneration
# So you transplant it like this:
vlm.model.language_model = llm.model
vlm.save_pretrained(...)

> [@zouYC2021](https://github.com/zouYC2021) Came across this bug today, after training a Qwen3.5 model with `trl`.
> ### Quick and dirty fix:
> 
> Replace new `config.json` produced after training with the one before that:
> 
> mv $TRL_CHECKPOINT_PATH/config.json $TRL_CHECKPOINT_PATH/config.json.old # backup 
> cp $ORIGINAL_CHECKPOINT_PATH/config.json $TRL_CHECKPOINT_PATH/config.json
> 
> Same for `tokenizer_config.json`.
> ### Wait for the bug to be really fixed in future `vllm` release

I tried this @zouYC2021 . I even copied the preprocessor_config.json from the original repo of huggingface. The model started loading. I was happy but then...

```
ValueError: Following weights were not initialized from checkpoint: {'visual.blocks.16.norm2.bias', 'visual.blocks.4.norm2.weight', ...
```

So I think it means when a model is saved as Text Model then it cannot be loaded with a Multimodal config. The error persists even if I remove the vision model kv pairs from the config.json.

Is there a solution? I use vllm 0.23.0 and transformer 5.12.1. And I use the llmcompressor 0.12.0 to get the W4A16 Qwen3.5 9B model. 
(1) The --language-model-only can not solve the error. 
(2) copy the preprocessor_config.json can not solve the error. 

Please try the solution suggested in my previous message

`

[rekey.py](https://github.com/user-attachments/files/29383665/rekey.py)

`


Use this code to transform your sft model.