Tachyeon/diffusion-asr

Diffusion ASR

Speech recognition with a block-diffusion language model as the decoder. A frozen audio encoder and a frozen text diffusion LM are joined by a small trained adapter plus LoRA — about 1 % of the stack is trained.

4.60 % WER on LibriSpeech test-clean (all 2,620 clips).

These are trained deltas, not a standalone model. This repo holds the 5.25 M-parameter adapter and the 17.4 M-parameter LoRA. Running them also needs the two frozen base models below and the inference code from ipritamdash/diffusion-asr-research.

Why this exists

The audio encoder came from a working ASR system. That system's own decoder is thrown away here and a text-only diffusion model put in its place, connected through roughly 5 M trained parameters.

The question is whether a language model that has never seen audio can be conditioned on acoustic evidence through a thin learned interface — and whether writing several tokens per forward pass buys anything.

Compare it against other frozen-encoder + frozen-LM + adapter systems. Do not compare it against end-to-end ASR trained on thousands of hours; it saw 30.

Results

Full LibriSpeech test-clean, 2,620 clips, corpus WER after Whisper English normalisation.

Decoder	Block width	WER	RTF	vs real time
Reference loop	2	4.60	—	—
Reference loop	4	5.23	—	—
LMDeploy engine	2	5.00	0.109	9.1×
LMDeploy engine	4	5.63	0.074	13.5×

RTF (real-time factor) is compute seconds ÷ audio seconds; lower is faster. Block width is a quality-versus-speed dial — widening from 2 to 4 costs about 0.6 WER and buys throughput. Both ends are usable.

The engine reads 0.40 worse than the reference loop at both block widths. A constant offset across two independent settings is not measurement variance; the cause has not been identified, and it is stated here rather than left out.

Error analysis

Computed over every clip of the 4.60 decode.

Output health — zero empty transcripts, zero repeat loops, zero length blow-ups, and no clip hit the token budget.

Composition — of 53,029 reference words: 3.75 % substituted, 0.40 % deleted, 0.45 % inserted. Four errors in five are a wrong word rather than a missing one; the model commits to a plausible word from unclear evidence instead of failing to produce one.

Substitutions — 40 % are within a small character edit of the reference (jailer → jailor, holmes → homes), which reads as blurred acoustic detail rather than a language failure.

Multi-piece words — words the tokeniser splits into several subword pieces are 7.5 % of words but carry 52 % of errors, at 13× the failure rate of single-piece words. This is the dominant failure mode.

Length and position — the shortest quarter of clips is hardest (5.85 % against 4.37 % on the longest), and quality is flat across an utterance. A fixed token budget and a block decoder could have degraded on long input; neither did.

Files

File	Contents	Params
adapter.pt	audio adapter + gain, CTC logit scale, CTC blank projection	5,250,049 (+2,050)
lora.pt	LoRA on the decoder, PEFT state dict, 392 tensors	17,432,576

adapter.pt keys: adapter, log_s, blank_w, blank_b, epoch, adapter_kind (conv), adapter_stride (1), attn_mode (stair), plus dev metrics. lora.pt keys: lora, epoch, dev_ce, objective.

log_s, blank_w and blank_b belong to the CTC alignment loss used during training. They are not used at decode time, and are kept so training can be resumed or audited.

Base models — frozen, download separately

Role	Repo	Pinned revision
Decoder	JetLM/SDAR-1.7B-Chat	97cf0dc9a7e85433f3f71a6bbac6fec9d381294f
Encoder	Qwen/Qwen3-ASR-0.6B	5eb144179a02acc5e5ba31e748d22b0cf3e303b0

Use these exact revisions — the weights here were trained against them.

The decoder must be loaded through the project repo's patched loader (qwen_sdar/sdar.py::load_patched_sdar), which swaps CUDA-only flash-attn for SDPA and adds the embeddings-input path. Plain from_pretrained will not reproduce these numbers.

Architecture

audio 16 kHz → Qwen3-ASR-0.6B encoder → adapter → SDAR-1.7B decoder + LoRA → text
               frozen, 1024-d @ 12.5 Hz  trained   frozen                     trained

Adapter — Conv1d(1024→1024, k=3, stride=1) → GELU → Linear(1024→2048) → LayerNorm(2048) → ×gain. Stride 1 keeps the encoder's native 12.5 Hz; pooling to 6.25 Hz measured worse across three seeds per arm, at roughly half the training cost.

gain initialises to the RMS of the decoder's token embeddings. A random projection emits vectors 10–50× the size of real word vectors, and a decoder fed a prefix that loud learns to ignore it; matching the embedding scale is what makes the audio legible at all.

LoRA — r=16, alpha=32, dropout 0, on q,k,v,o,gate,up,down in all 28 layers. LoRA scales updates by alpha/r, so the effective learning rate on those weights is 2× nominal.

Attention is block-causal at every stage, training and inference alike. Each block sees the audio, all earlier blocks, and itself. Training and serving therefore use the same convention, and the model runs on a standard diffusion inference engine without retraining.

Training

30 hours of LibriSpeech train-clean-100, speaker-disjoint from the held-out split. Three stages, never joint:

Stage	Trains	Loss
Grounding	adapter	CTC only, decoder detached
SFT	adapter	CTC + diffusion cross-entropy
LoRA	LoRA	diffusion cross-entropy only, adapter frozen

Roughly 32 GPU-hours in total.

Decoding

Block-causal, blocks written left to right, earlier blocks frozen once committed.

Block width	2 (best) or 4 (faster)
Denoising steps per block	up to 4
Commit threshold	0.95
If no position clears it	commit the most confident one, so the loop always advances
Token budget	160

Block width 2 is below the 4–64 range the SDAR paper sweeps, and below the width this checkpoint trained at. It measured better anyway, on both decoders.

Usage

The encoder and decoder have conflicting dependencies, so inference runs in two steps in two environments. Both need a CUDA GPU.

git clone https://github.com/ipritamdash/diffusion-asr-research
cd diffusion-asr-research

huggingface-cli download Tachyeon/diffusion-asr adapter.pt lora.pt --local-dir .

# 1. encoder environment: audio -> features
pip install -r requirements-encoder.txt
python featurize.py --audio clip.wav --out features.pt

# 2. decoder environment: features -> transcript
pip install -r requirements-decode.txt
python decode.py --features features.pt --adapter adapter.pt --lora lora.pt --block_length 2

decode.py reads adapter_stride from the checkpoint and refuses to run if it is not recorded. A stride-1 and a stride-2 adapter have identical tensor shapes, so guessing wrong loads cleanly and silently emits a prefix of the wrong length.

For serving, merge the LoRA and run on LMDeploy's PyTorch diffusion engine; see engine/ in the project repo.

Scope

A research model for English read speech. LibriSpeech is clean, read audiobook material and these numbers describe that domain.

Licence

The adapter and LoRA here are trained deltas. Any use depends on the two frozen base models, which carry their own licences — check JetLM/SDAR-1.7B-Chat and Qwen/Qwen3-ASR-0.6B before redistributing anything, and do not redistribute a merged checkpoint (which embeds the base weights) without confirming their terms.