HASOC Hindi-English Code-Mixed Hate Speech Detector (Student)
DistilBERT-multilingual fine-tuned on the same HASOC 2021 code-mixed hate speech data as the teacher model, intended as a smaller/faster alternative.
Model Details
- Base model: distilbert-base-multilingual-cased
- Task: Binary classification (HOF = Hate/Offensive, NOT = Not offensive)
- Parameters: 135M (51.3% smaller than the teacher's 278M)
Evaluation Results
- Macro-F1 (test set): 70.80% (96.1% of teacher's performance)
- Inference latency: 23.5ms vs teacher's 96.4ms (4.1x faster)
Note on Training Method
This model was trained via direct fine-tuning on hard labels, not knowledge distillation. Multiple knowledge distillation configurations (soft-label KL-divergence loss combined with hard-label loss, at various temperature/alpha settings) were tested and found not to outperform plain fine-tuning for this teacher/task combination — likely because the teacher's errors are systematic (biased toward certain vocabulary) rather than random, so softening its output amplified those biases rather than adding useful signal.
Limitations
Shares similar failure modes to the teacher (over-triggering on charged vocabulary, missing subtle coded hate), though error overlap analysis found the two models' mistakes are only ~70% shared — the student independently corrects roughly 30% of the teacher's errors on both false positives and false negatives.
Project Repository
Full code, error analysis, distillation experiments, and benchmarking are documented in the GitHub repository.
Demo
Try both models side by side in the live Gradio demo.
- Downloads last month
- -