The Whisper-Small is fine tuned on 20,000 urdu audio samples. WER is reduced from 97% to 55%
This project is licensed under the MIT License. It utilizes the https://huggingface.co/datasets/Shzaib/SpecDox-Dataset dataset, which is licensed under the Apache License 2.0. A copy of the Apache 2.0 License can be found in LICENSE-DATASET.md
Files Description:
adapter_config.json: Tells the peft library how to attach the adapter to the base Whisper model.
It contains essential metadata such as your LoRA rank (r=32), scaling factor (alpha=64), target attention modules (q_proj, v_proj), dropout rate, and the base model name (openai/whisper-small)
adapter_model.safetensors: Contains the low-rank mathematical matrices updated during training on Urdu-to-English dataset.
preprocessor_config.json: Instructs the WhisperFeatureExtractor on how to convert raw 16 kHz audio waveforms into 80-channel log-mel spectrograms.
It stores parameters like sampling rate (16000), FFT window size, hop length, and feature dimensions so input audio matches the exact tensor structure the model was trained on.
tokenizer_config.json: Configures the WhisperTokenizer to translate raw model output IDs back into human-readable text. It specifies special prompt tokens (such as <|startoftranscript|>,
<|en|>, and <|translate|>), padding tokens, and vocabulary decoding rules for English output generation.
How to use Fine Tuned Model
Import Libraries
from peft import PeftConfig, PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor
Specify Hugging Face Repository ID
peft_model_id = "Aimen20/Whisper-Small-FineTuned"
Check configuration of LORA adapter
peft_config = PeftConfig.from_pretrained(peft_model_id)
Pass configuration to processor
processor = WhisperProcessor.from_pretrained(
peft_config.base_model_name_or_path, language="English", task="translate"
)
Get base model
base_model = WhisperForConditionalGeneration.from_pretrained(
peft_config.base_model_name_or_path
)
Attach LORA adapter to base model
model = PeftModel.from_pretrained(base_model, peft_model_id)