---
language: ja
license: apache-2.0
tags:
- qwen
- qwen3
- merged-model
- sft
- cot
- structured-output
- japanese
- math
pipeline_tag: text-generation
datasets:
- daichira/structured-3k-mix-sft
base_model:
- azuki-digital/qwen3-4b-struct-lora-v4-merged
---

# qwen3-4b-struct-lora-v11-merged

This repository provides a **LoRA-merged base model** derived from:

azuki-digital/qwen3-4b-struct-lora-v4-merged

> This is **NOT** a LoRA adapter.  
> This is a fully merged, standalone model.

---

## What is this model?

This model is a **structured-output specialized base model**.

It was created by:

1. Fine-tuning Qwen3-4B-Instruct with LoRA for structured outputs
2. Merging the LoRA weights into the base model
3. Publishing the merged result as a new foundation checkpoint

This allows future LoRA training to start from a **better structured-output prior**.

## Why this merged model exists

Typical workflow:

Qwen3-4B-Instruct

↓ LoRA (structured output training)

↓ merge

qwen3-4b-struct-lora-v4-merged

↓ new LoRA training (v11)

This model significantly stabilizes later SFT and improves convergence.

---

## Training Configuration

| Item | Value |
|---|---|
| Base model | azuki-digital/qwen3-4b-struct-lora-v4-merged |
| Method | LoRA SFT (no quantization, bf16) |
| Max sequence length | **4096** |
| Epochs | 2 |
| Learning rate | 1e-5 |
| Warmup ratio | 0.05 |
| Weight decay | 0.05 |
| LoRA r | 32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.05 |
| Target modules | q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj |
| Mask CoT | Yes (`after_marker`) |
| Dataset | daichira/structured-3k-mix-sft |

---

## Usage

This is a fully merged standalone model. No LoRA adapter is required.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "azuki-digital/qwen3-4b-struct-lora-v11-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
```

---

## License & Compliance

This model inherits the license of:

- azuki-digital/qwen3-4b-struct-lora-v4-merged (base model)
- Structured-output dataset used during original LoRA training

Users must comply with the original base model terms.