SpeechX

Multi-task speech classifier based on the Whisper-base encoder that predicts a speaker's gender and language from a short audio clip.

Use cases

• Fast language identification (LID) — 19 ms per clip end-to-end on a single GPU (well under 40–50 ms), or 270 clips/s batched; 109 languages, 97% top-5 accuracy
• Gender detection from voice — 97.8% accuracy, same single forward pass (both predictions come together)
• Pre-routing for ASR pipelines (pick the right transcription model per clip), call-center analytics, voice assistant profiling, dataset auditing/labeling

Model architecture

audio (16 kHz mono, short clips)
  → Whisper log-mel feature extractor (80 bins)
  → Whisper-base encoder (20.6M params, fp16)
  → mean pooling over time → 512-d embedding
  → Linear(512 → 2)    gender head
  → Linear(512 → 109)  language head

The Whisper decoder is not used. Weights are stored in fp16 (39 MB).

Training data

• Train: 423,772 clips (106 languages)
• Validation: 50,000 clips (108 languages, stratified)

Training: 3 epochs, AdamW, bf16, OneCycle LR (heads 1e-4, encoder 1e-5).

Results (validation, 50K clips)

Gender (2 classes)

Metric	Value
Accuracy	97.75%
Precision (macro)	97.52%
Recall (macro)	97.63%
F1 (macro)	97.58%

Class	Precision	Recall	F1
female	0.967	0.972	0.969
male	0.984	0.981	0.982

Language (109 classes)

Metric	Value
Top-1 accuracy	87.16%
Top-1 accuracy (66 well-resourced langs, F1 ≥ 70%)	89.96%
Top-5 accuracy	97.10%
Precision (macro)	72.68%
Recall (macro)	67.77%
F1 (macro)	68.66%

Key languages (validation F1):

Language	Precision	Recall	F1
Arabic (ar)	0.904	0.896	0.900
English (en)	0.869	0.859	0.864
Spanish (es)	0.834	0.848	0.841
French (fr)	0.892	0.829	0.859
German (de)	0.856	0.859	0.858
Italian (it)	0.816	0.873	0.844
Portuguese (pt)	0.735	0.862	0.794
Russian (ru)	0.783	0.830	0.806
Turkish (tr)	0.828	0.877	0.852
Persian (fa)	0.793	0.875	0.832
Urdu (ur)	0.821	0.858	0.839
Hindi (hi)	0.671	0.647	0.659
Japanese (ja)	0.955	0.943	0.949
Korean (ko)	0.933	0.949	0.941
Chinese Mandarin (zh-CN)	0.876	0.884	0.880
Chinese Taiwan (zh-TW)	0.800	0.823	0.811
Indonesian (id)	0.900	0.883	0.892
Swahili (sw)	0.918	0.938	0.928
Polish (pl)	0.814	0.836	0.825
Ukrainian (uk)	0.843	0.858	0.851

High-resource languages overall (ta, ka, hu, lg, rw…) reach 90–97% F1. Languages with < 50 training clips are unreliable.

Latency

Device	Mode	Latency
NVIDIA L4 (fp16)	single clip (load + features + forward)	18.6 ms (54 clips/s)
NVIDIA L4 (fp16)	batch 64	237 ms/batch → 270 clips/s
CPU (8 threads, fp32)	single clip	317 ms (3.2 clips/s)

GPU is fast enough for real-time LID/gender gating with a sub-50 ms budget per clip. CPU works for offline/batch processing; for real-time CPU use consider int8 quantization or distilling to whisper-tiny.

Usage

SpeechX is a native HF model (auto_map + trust_remote_code) — load it with AutoModel:

import torch
import torchaudio
from transformers import AutoModel, WhisperFeatureExtractor

model = AutoModel.from_pretrained(
    "AhmedReda25300/SpeechX", trust_remote_code=True, dtype=torch.float16
).cuda().eval()
fe = WhisperFeatureExtractor.from_pretrained("AhmedReda25300/SpeechX")

wav, sr = torchaudio.load("clip.wav")
wav = wav.mean(dim=0)                                    # mono
if sr != 16000:
    wav = torchaudio.functional.resample(wav, sr, 16000)

x = fe(wav.numpy(), sampling_rate=16000, return_tensors="pt").input_features.cuda().half()

with torch.no_grad():
    pred = model.predict(x)                              # convenience method
print(pred)
# {'gender': 'male', 'gender_confidence': 0.9571,
#  'language_topk': [{'lang': 'ar', 'prob': 0.312}, {'lang': 'fa', 'prob': 0.184}, ...]}

License

Apache-2.0. The Whisper base model is MIT-licensed (OpenAI); Common Voice audio is CC0.