dakshp2005/whisper-medium-indic

Whisper Medium — Indic (fine-tuned)

dakshp2005/whisper-medium-indic is openai/whisper-medium fine-tuned for automatic speech recognition (speech → text) in [TODO: language(s), e.g. Hindi and Gujarati]. It transcribes speech in the same language and script it was spoken in. It is not meant for translation.

Quick facts

Property	Value
Task	Automatic speech recognition (transcription)
Base model	openai/whisper-medium
Architecture	Whisper encoder–decoder (WhisperForConditionalGeneration), ~769M parameters
Weights format	Safetensors, float32
Language(s)	[TODO: e.g. Hindi (hi)]
Output script	[TODO: e.g. Devanagari]
Whisper language setting	[TODO: e.g. "hindi"]
Whisper task setting	"transcribe"
Input audio	16 kHz, mono, float waveform
Max input per forward pass	30 seconds (longer audio must be chunked)
Training data domain	[TODO: e.g. read speech, conversational, phone calls, news]
Fine-tuned on	Kaggle (notebook)

Intended use

• Transcribing [TODO: language(s)] speech to text, such as recordings, voice notes, calls and meetings.
• Testing and prototyping real-time captioning using chunked or streaming inference (see the notes below).

Not intended for: translation, speaker identification, languages outside those listed above, or high-stakes uses such as medical or legal records without human review.

How to use

Transformers

import torch
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

asr = pipeline(
    "automatic-speech-recognition",
    model="dakshp2005/whisper-medium-indic",
    torch_dtype=dtype,
    device=device,
)

result = asr(
    "audio.wav",
    chunk_length_s=30,  # needed for audio longer than 30 s
    generate_kwargs={"language": "TODO_LANGUAGE", "task": "transcribe"},  # e.g. "hindi"
)
print(result["text"])

Always pass language and task explicitly. Otherwise Whisper may auto-detect the wrong language or translate into English.

faster-whisper (recommended for CPU and real-time use)

Convert the model to CTranslate2 once:

pip install faster-whisper ctranslate2 transformers
ct2-transformers-converter \
  --model dakshp2005/whisper-medium-indic \
  --output_dir whisper-medium-indic-ct2 \
  --copy_files tokenizer.json preprocessor_config.json \
  --quantization int8        # use float16 on an NVIDIA GPU

Then run it:

from faster_whisper import WhisperModel

model = WhisperModel("whisper-medium-indic-ct2", device="cpu", compute_type="int8")
segments, info = model.transcribe(
    "audio.wav",
    language="TODO_CODE",   # ISO code, e.g. "hi"
    task="transcribe",
    beam_size=1,
    vad_filter=True,
)
for s in segments:
    print(f"[{s.start:.1f}s → {s.end:.1f}s] {s.text}")

Training details

• Base model: openai/whisper-medium
• Training data: [TODO: dataset name(s), link(s), and total hours of audio per language]
• Data preprocessing: [TODO: e.g. resampled to 16 kHz, removed clips over 30 s, text normalization applied]
• Hardware: Kaggle [TODO: GPU type, e.g. 2× T4 or P100]
• Training hyperparameters:
  • learning rate: [TODO]
  • batch size (per device × gradient accumulation): [TODO]
  • max steps / epochs: [TODO]
  • warmup steps: [TODO]
  • precision: [TODO: e.g. fp16 mixed precision]
  • optimizer: [TODO: e.g. AdamW]
• Training framework: Hugging Face Transformers Seq2SeqTrainer, transformers [TODO: version]

Evaluation

Model	Test set	WER ↓
openai/whisper-medium (base)	[TODO]	[TODO]
dakshp2005/whisper-medium-indic	[TODO]	[TODO]

[TODO: say how WER was computed, e.g. with or without text normalization, and on which split.]

Limitations

• 30-second window. Whisper processes at most 30 s at a time, so longer audio must be chunked.
• Not natively streaming. Real-time use requires buffering, voice activity detection (VAD) and repeated re-transcription of the current buffer (see below).
• Hallucinations. Whisper can produce text on silence, music or noise, or repeat phrases. Use VAD and filter out low-confidence or no_speech_prob-flagged segments.
• Telephone audio (8 kHz). [TODO: say whether the training data included phone-quality audio. If not, expect lower accuracy on call recordings.]
• Code-mixing. [TODO: describe how the model handles mixed speech such as Hindi–English. Is English written in Latin script or transliterated?]
• Accuracy may drop for accents, dialects, noisy environments and domains not covered by the training data.

Notes for developers and AI coding agents

These are the key facts needed to integrate this model correctly:

1. Load it as a Transformers Whisper model: WhisperForConditionalGeneration plus WhisperProcessor (or AutoModelForSpeechSeq2Seq / AutoProcessor). No trust_remote_code is needed.
2. Set the language and task at generation time: language="TODO_LANGUAGE", task="transcribe". Do not rely on forced_decoder_ids in the config. If a newer Transformers version raises a conflict about forced_decoder_ids, clear it at load time (model.generation_config.forced_decoder_ids = None) and pass language and task instead.
3. Input must be 16 kHz mono float32 audio, at most 30 s per forward pass.
4. For real-time transcription:
   • Stream audio in 100–250 ms chunks and segment it with VAD (e.g. Silero VAD).
   • Re-transcribe the active speech buffer every ~1 s for partial results.
   • Finalize a segment after ~500–700 ms of silence, or when the buffer reaches ~25 s.
5. Recommended engine: faster-whisper (CTranslate2), using int8 on CPU or float16 on GPU. Expect partial-result latency of roughly 2–5 s on a typical laptop CPU, and under ~1 s on a GPU such as a T4.
6. The weights are float32, about 3 GB. Convert to float16 or int8 for faster inference.

Citation

If you use this model, please cite the original Whisper paper:

@misc{radford2022whisper,
  title  = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  year   = {2022},
  eprint = {2212.04356},
  archivePrefix = {arXiv}
}

Author

Fine-tuned by dakshp2005.