Khon198/indictts-deepfake-detector

Model Card: IndicTTS Deepfake Speech Detector

Model Details

• Base model: ntu-spml/distilhubert
• Architecture: HubertForSequenceClassification — DistilHuBERT encoder (frozen CNN feature extractor + fine-tuned transformer layers) with a linear binary classification head
• Task: Binary audio classification — authentic human speech vs. AI-generated (TTS) speech
• Language(s): Multilingual, Indic languages (see Training Data)
• License: Inherits ntu-spml/distilhubert's license (Apache 2.0) for the base model; fine-tuned weights license is up to you to set
• Fine-tuned by: [your name/org]
• Fine-tuning date: [fill in]

Intended Use

Intended for: demonstrating automated detection of synthetic speech for educational/portfolio purposes, and as a starting point for further research into speech deepfake detection.

Not intended for: any decision with real-world consequences for a person — content moderation, legal/forensic evidence, authentication systems, or any setting where a false result could cause harm. See Limitations below.

Training Data

SherryT997/IndicTTS-Deepfake-Challenge-Data — Indic-language speech labeled is_tts (0 = authentic, 1 = synthetic).

• Trained on a 10,000-sample random subset (seed=37) of the full ~16GB training split, with a 90/10 train/validation split (seed=42)
• Audio standardized to 16kHz mono, 2-second (32,000 sample) windows, trimmed or zero-padded as needed
• Class balance of the training subset: [fill in — see debug_check.py's printed class balance line]

Training Procedure

Optimizer	AdamW
Learning rate	1e-4, 1,000 warmup steps
Weight decay	0.005
Epochs	7 (15,750 steps total)
Batch size	4
Precision	fp32
Feature encoder	Frozen
Model selection	Best checkpoint by validation loss

Evaluation

Held-out internal validation split (10% of the 10k training subsample), best checkpoint by validation loss (step 8,500 of 15,750):

Metric	Score
Accuracy	97.30%
Precision	97.44%
Recall	96.82%
F1	97.13%
ROC-AUC	0.9960

Scored 0.9761 on the IndicTTS Deepfake Challenge public leaderboard (competition test set, disjoint from training data).

Limitations

• Training scale: fine-tuned on 10k samples out of a much larger dataset. Performance on the full data distribution, and on TTS systems/languages underrepresented in the subset, is unverified.
• Fixed input length: every clip is trimmed or padded to exactly 2 seconds. Longer clips are truncated rather than analyzed in full; very short clips are zero-padded, which may reduce reliability on edge cases.
• Distribution shift: trained and validated on audio from a single benchmark dataset. Real-world audio (background noise, compression artifacts, different microphones, TTS systems not represented in training) will likely perform worse than the reported metrics — observed anecdotally during demo testing, where out-of-distribution non-speech audio was classified with high but not necessarily meaningful confidence.
• Confidence calibration: the model was trained with cross-entropy loss and no explicit calibration step. High reported confidence (e.g. 99–100%) should not be read as a calibrated probability of correctness, especially on inputs unlike the training distribution.
• No quantization/compression applied — served as the raw fp32 fine-tuned checkpoint. See main README's "Known limitations" for planned optimization work.

Ethical Considerations

Speech deepfake detectors can be used both defensively (flagging synthetic media, protecting against voice-cloning fraud) and, if inverted, to help adversaries iterate against detection systems. This model is released for educational/demonstration purposes with modest training scale and unverified real-world robustness — it should not be relied on as a production safeguard against audio fraud without substantially more rigorous evaluation, adversarial testing, and domain-specific validation.

How to Use

from transformers import AutoModelForAudioClassification, AutoFeatureExtractor

model = AutoModelForAudioClassification.from_pretrained("path/to/model")
feature_extractor = AutoFeatureExtractor.from_pretrained("path/to/model")

See backend/predict.py in the main repo for the full inference pipeline (audio decoding, resampling, and classification).