whisper-small-sinhala-run15-backup

Fine-tuned openai/whisper-small for Sinhala (si) ASR with mixed-data anti-catastrophic-forgetting training.

This is the new project best model, beating run12 on the Sinhala test set while dramatically reducing English catastrophic forgetting.

---

Results

Sinhala ASR (stratified_v6 test set, 15,860 samples)

Metric	Value
Test WER	15.48%
Test CER	4.24%
Val WER (epoch 6)	18.60%
Val CER (epoch 6)	5.87%

English Catastrophic Forgetting (LibriSpeech test-clean, 2,620 samples)

Model	English WER	Delta vs base	Severity
whisper-small base	4.27%	—	—
run12 (pure Sinhala)	98.70%	+94.43 pts	severe
run15 (this model)	8.00%	+3.73 pts	mild

The 7.5% English mix in training reduced catastrophic forgetting from +94 pts to +3.73 pts — a 96% reduction.

---

Training Details

Parameter	Value
Base model	openai/whisper-small
Dataset	stratified_v7 (mixed)
Train set	133,205 rows — 92.5% Sinhala + 7.5% English
Val/Test set	Sinhala-only (v6 split, comparable across runs)
Learning rate	8.56e-5
LR scheduler	cosine
Batch size	64
Epochs	6
Warmup steps	500
Noise aug prob	0.2
Stretch aug prob	0.2
Pitch aug prob	0.2
Total steps	12,492

Validation WER per Epoch

Epoch	Val WER	Val CER
1	31.64%	9.78%
2	25.62%	7.95%
3	22.42%	6.90%
4	20.41%	6.32%
5	18.75%	5.91%
6	18.60%	5.87%

---

Weight Delta Analysis (run15 vs run12)

Per-layer L2 norm of (run15 − run12) relative to run12 base norm, showing which parts of the model changed most due to the English data mix.

Most Changed Layers

Layer	Relative Change
decoder.embed_positions	60.0%
decoder.embed_tokens	46.1%
decoder.layers.*.encoder_attn.out_proj	39–49%
decoder.layers.*.encoder_attn.{k,v}_proj	40–45%
decoder.layers.*.self_attn.out_proj	39–45%

Least Changed Layers

Layer	Relative Change
encoder.layers.*.final_layer_norm.{weight,bias}	0.09–0.16%

Key insight: The decoder changed dramatically (40–60% in cross-attention and embeddings) while the encoder barely moved (0.1% in layer norms). The English mix kept the decoder's cross-attention and positional embeddings anchored to English representations, while the encoder learned Sinhala acoustics similarly to run12. This explains why English forgetting dropped so dramatically — the decoder retained its English language modelling capability.

---

Anti-Forgetting Strategy

Previous runs (run12, run14) used pure Sinhala training data and showed severe catastrophic forgetting (+94–96 pts English WER delta). This run adds 10,000 LibriSpeech train-clean-100 English samples (7.5% of training data) to the v6 Sinhala training set. Each row carries a language column ("si" / "en") and is tokenized with the correct language prefix token (<|si|> / <|en|>) during training.

---

Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

model = WhisperForConditionalGeneration.from_pretrained("Yohan2003/whisper-small-sinhala-run15-backup")
processor = WhisperProcessor.from_pretrained("Yohan2003/whisper-small-sinhala-run15-backup")

# For Sinhala transcription
model.generation_config.language = "sinhalese"
model.generation_config.task = "transcribe"