Qwen3-ASR-0.6B-TW-Agent (joint S2T v6)

LoRA adapter turning Qwen/Qwen3-ASR-0.6B into a Taiwan-localized speech-to-action agent (built for the office phone-attendant use case): it transcribes zh-TW/English speech and acts on it in the same decoder pass — no separate ASR + LLM pipeline.

The audio encoder is frozen; only decoder LoRA weights (r32) are trained, jointly on zh-TW ASR (Common Voice 17), English ASR (LibriSpeech train-clean-100), telephone-domain dialogs, and ~54k function-calling examples (50/50 zh/en, 13% "decline to call" negatives).

Output format: transcript-first

Every response to an audio turn starts with a verbatim transcript, then the action. The transcript is not decoration — the closing </transcript> gives the model a hard stop between hearing and acting (separately-trained adapters rambled without it), and it gives the calling application the raw ASR text for logging, confirmation prompts, or correction UIs.

1 — Caller's request maps to a tool (Hermes-format <tool_call>, one JSON object per call):

<transcript>請幫我轉接行銷部的陳怡君</transcript>
<tool_call>
{"name": "search_contacts", "arguments": {"query": "陳怡君", "department": "行銷"}}
</tool_call>

2 — No tool fits (trained to decline rather than force a call — see the irrelevance scores):

<transcript>你們公司幾點下班？</transcript>
我們的上班時間是週一至週五九點到六點。

3 — Text-only turns (tool results, follow-up questions) skip the transcript and reply or call tools directly, following the standard Qwen3 chat template with tools=[...].

Parsing: take everything between <transcript>...</transcript> as the ASR result, then scan for <tool_call>...</tool_call> blocks (JSON inside); anything else is the spoken reply.

Benchmarks (2026-07-30)

Tool-calling — BFCL v4, 11 non-agentic categories (no java/js), limit 150/cat, official ast_checker; TW-BFCL v4 is a zh-TW localization of the same categories:

model	size	BFCL v4 (en)	TW-BFCL v4 (zh-TW)
this model (v6)	0.6B	81.5	67.5
LFM2.5-1.2B-Instruct	1.2B	76.4	56.9
Qwen3-0.6B	0.6B	73.9	59.9
LFM2.5-350M	0.35B	65.3	46.0
Qwen3.5-0.8B	0.8B	36.9	23.7
FunctionGemma-270M	0.27B	28.0	19.0

ASR — 300 Common Voice 17 zh-TW test clips (CER, OpenCC s2tw-normalized) and 200 LibriSpeech test-clean clips (WER, case-insensitive), identical pipeline for all rows:

model	size	zh-TW CER	en WER
this model (v6)	0.6B	4.91	11.56
Qwen3-ASR-0.6B (base)	0.6B	5.17	3.50
whisper-small	0.24B	10.64	3.52
whisper-base	0.07B	19.15	4.78

Honest caveats:

• English ASR regresses vs the base model (11.56 vs 3.50 WER). The fine-tune trades English transcription accuracy for tool-calling and zh-TW gains; English WER plateaus at ~11.5 regardless of how much English audio is mixed in (5k and 10k rows tested).
• TW-BFCL numbers are not comparable to English-BFCL leaderboard numbers: its gold answers are augmented with zh alternatives (see the dataset card) and ~29% of AST rows are flagged _solvable=false (cross-lingual entity mismatch). On the _solvable subset this model scores 68.7.
• live_relevance (n=16) dropped to 75% after training the model to decline more readily.

Usage (speech → tool call)

from qwen_asr.core.transformers_backend.modeling_qwen3_asr import Qwen3ASRForConditionalGeneration
from qwen_asr.core.transformers_backend.processing_qwen3_asr import Qwen3ASRProcessor
from peft import PeftModel
import torch

base = "Qwen/Qwen3-ASR-0.6B"
proc = Qwen3ASRProcessor.from_pretrained(base)
thinker = Qwen3ASRForConditionalGeneration.from_pretrained(base, dtype=torch.bfloat16).thinker
model = PeftModel.from_pretrained(thinker, "<this repo>").cuda().eval()
# render your system prompt + tools + <|audio_pad|> user turn with the processor, then generate;
# the reply is <transcript>...</transcript> followed by a Hermes <tool_call> or a text answer.

For text-only use (e.g. BFCL harnesses), merge the adapter into a plain Qwen3ForCausalLM (the audio tower is not needed).

Trained with LoRA r32 on a frozen-encoder Qwen3-ASR-0.6B; ~81.5k-row multi-task mix, 1 epoch, lr 5e-5, bf16.