qwen3-4b-struct-lora-v11-merged

This repository provides a LoRA-merged base model derived from:

azuki-digital/qwen3-4b-struct-lora-v4-merged

This is NOT a LoRA adapter.
This is a fully merged, standalone model.

---

What is this model?

This model is a structured-output specialized base model.

It was created by:

1. Fine-tuning Qwen3-4B-Instruct with LoRA for structured outputs
2. Merging the LoRA weights into the base model
3. Publishing the merged result as a new foundation checkpoint

This allows future LoRA training to start from a better structured-output prior.

Why this merged model exists

Typical workflow:

Qwen3-4B-Instruct

↓ LoRA (structured output training)

↓ merge

qwen3-4b-struct-lora-v4-merged

↓ new LoRA training (v11)

This model significantly stabilizes later SFT and improves convergence.

---

Training Configuration

Item	Value
Base model	azuki-digital/qwen3-4b-struct-lora-v4-merged
Method	LoRA SFT (no quantization, bf16)
Max sequence length	4096
Epochs	2
Learning rate	1e-5
Warmup ratio	0.05
Weight decay	0.05
LoRA r	32
LoRA alpha	64
LoRA dropout	0.05
Target modules	q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
Mask CoT	Yes (after_marker)
Dataset	daichira/structured-3k-mix-sft

---

Usage

This is a fully merged standalone model. No LoRA adapter is required.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "azuki-digital/qwen3-4b-struct-lora-v11-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

---

License & Compliance

This model inherits the license of:

• azuki-digital/qwen3-4b-struct-lora-v4-merged (base model)
• Structured-output dataset used during original LoRA training

Users must comply with the original base model terms.