lapwing-small: pronunciation assessment and transcription from whisper-base

This is a fine-tune of openai/whisper-base for pronunciation assessment of English learners. One encoder pass over a 16 kHz mono recording gives:

1. Phone posteriors: log_probs [T, 40] at 20 ms frames (CTC: blank + 39 ARPAbet phones, no stress), for goodness-of-pronunciation (GOP) scoring;
2. Cross-attention keys/values for the stock whisper-base decoder, which gives a transcript for "did they read the right sentence" checks only.

It is the small option. Its sibling GersonHMG/lapwing is more accurate but four times larger. (Internal name: base-short.)

Parameters	77.2M: encoder + phone head 28.4M, decoder 48.9M
Changes to the base	LoRA (rank 16, α 32) on q/v of the top 3 of 6 encoder layers, merged; new phone head
Phone head	softmax layer mix of 7 hidden states → LayerNorm → BiLSTM 2 × 320 → Linear 640→40 (4.7M)
Runtime size	168 MB (int8 ONNX)
Latency, 5 s clip, 1 CPU thread	0.32 s
Max input	30 s

Files

File	What it is
model.int8.onnx	samples [1,N] → log_probs [1,T,40], n_layer_cross_k/v [6,1,1500,512]
decoder.int8.onnx	stock whisper-base decoder (sherpa-onnx export format, self-attention KV cache)
tokens.txt	phone table
whisper-tokens.txt	Whisper vocabulary (base64 bytes → id)
whisper_meta.json	special token ids, decoder dims
gate.json	gate floor and its calibration statistics
model.safetensors	the fp32 weights of both graphs, for reference and parameter counting. Tensor names are the ONNX initializer names, prefixed model/ or decoder/. The runtime does not need this file.
SHA256SUMS	checksums

Phone ids: <blk> 0, AA 1, AE 2, AH 3, AO 4, AW 5, AY 6, B 7, CH 8, D 9, DH 10, EH 11, ER 12, EY 13, F 14, G 15, HH 16, IH 17, IY 18, JH 19, K 20, L 21, M 22, N 23, NG 24, OW 25, OY 26, P 27, R 28, S 29, SH 30, T 31, TH 32, UH 33, UW 34, V 35, W 36, Y 37, Z 38, ZH 39.

Architecture

samples [1,N] ─► loudness normalisation ─► Whisper log-mel, 80 bins, at the real length (no 30 s padding)
              ─► whisper-base encoder (2 conv + 6 transformer layers, d 512, 20 ms frames)
                   ├─ 7 hidden states ─► softmax-weighted sum ─► LayerNorm ─► BiLSTM 2×320 ─► Linear ─► log_probs
                   └─ last state + constant 30 s "silence tail" to 1500 frames ─► cross k/v for the decoder

Whisper's decoder was trained only on 30 s windows, so the graph appends the encoder output of silence after the real frames. Without it the decoder never stops and repeats the sentence.

Usage notes

• Input: float32 [1, N], 16 kHz mono, range [−1, 1]. Loudness and mel features are computed inside the graph.
• Requires onnxruntime ≥ 1.20. Tested with 1.23 and 1.28, CPU.
• Text decoding: greedy Whisper with prompt [50258, 50259, 50359, 50363]. Suppress no_timestamps, sot, no_speech, translate (and eot and token 220 at step 0). Stop at <|endoftext|>, when the output starts repeating a block, or at max(8, 6·seconds) + 8 tokens.
• Gate: reject an attempt when the CTC log-likelihood of the target phones per frame, L(c)/T, is below −0.374 (gate.json). It was fitted to accept 2.3% of wrong-sentence attempts.
• GOP colour thresholds are not calibrated. Only the gate and word ranking (which word is worst) were validated.

Results

Speaker-disjoint split; the numbers are on the held-out half. "L2-B" is an L2-ARCTIC sentence group no model trained on.

Metric	lapwing-small
Phone error rate, L2-ARCTIC B / all	20.7% / 19.1%
Mispronunciation detection F1 (P / R)	0.505 (0.49 / 0.52)
GOP phone AUC, L2-B	0.826
Correlation with expert scores (speechocean762), phone / word	0.493 / 0.410
Word ranking AUC, speechocean762 / L2-B	0.912 / 0.764
Phone error rate, native read speech (LibriSpeech)	4.9%
Text WER LibriSpeech / speechocean762 / L2-ARCTIC	6.2% / 35.0% / 16.2%

The int8 ONNX file scores within about 1 point of the PyTorch model on phone error rate and WER.

Training data

• LibriSpeech train-clean-100 (CC BY 4.0)
• L2-ARCTIC (CC BY-NC 4.0): Arabic, Mandarin, Hindi, Korean, Spanish and Vietnamese first languages
• speechocean762 (CC BY 4.0): Mandarin first language, partly children

Read speech only. Expect weaker results on spontaneous speech, other accents, noise and far-field audio.

Licence and attribution

• Non-commercial: CC BY-NC 4.0. The phone head and LoRA were trained on L2-ARCTIC, which is CC BY-NC 4.0. Get legal advice before any commercial use.
• Built on OpenAI's whisper-base (MIT).