news-classification-minilm

A SetFit few-shot text classifier that sorts short Arabic/English social-media-style text into four categories: news, historical/scientific facts, medical information, and non-news (personal/casual posts).

It's built for bilingual, mixed-register text — Modern Standard Arabic, Egyptian colloquial Arabic, and English, in both formal and casual tones — which is what makes it different from classifiers trained only on clean news headlines.

Label Mapping

ID	Label	Description	Example
0	news	Current-events reporting, official announcements	"عاجل: مجلس الوزراء يوافق على زيادة الحد الأدنى للأجور"
1	historical_scientific	Historical facts, science facts/trivia, "did you know" style	"TIL that octopuses have three hearts"
2	medical	Health information, symptoms, medical advice	"أعراض السكري تشمل العطش الشديد وكثرة التبول"
3	non_news	Personal posts, opinions, casual chat, ads	"Just finished my exams finally!! So relieved 🎉"

Model Details

• Model Type: SetFit (Sentence Transformer fine-tuned with contrastive learning + a LogisticRegression classification head)
• Sentence Transformer body: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Classification head: LogisticRegression
• Number of Classes: 4
• Languages: Arabic (MSA + Egyptian colloquial), English
• Maximum Sequence Length: 128 tokens

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Uses

Direct Use for Inference

pip install setfit

from setfit import SetFitModel

model = SetFitModel.from_pretrained("darck-12/news-classification-minilm")

label_names = {0: "news", 1: "historical_scientific", 2: "medical", 3: "non_news"}

texts = [
    "الحكومة أعلنت اليوم عن رفع أسعار الوقود",
    "just found out that skipping breakfast regularly might affect your metabolism",
]
preds = model.predict(texts)
for text, pred in zip(texts, preds):
    print(label_names[int(pred)], "-", text)

Intended Use

• Triaging or auto-tagging short-form bilingual (Arabic/English) social posts or messages into broad content categories.
• Filtering "news" vs. "non-news" content in a moderation or feed-ranking pipeline where Egyptian-dialect Arabic is expected.

Out-of-Scope Use

• Not validated on Arabic dialects other than Egyptian (e.g., Gulf, Levantine, Maghrebi) — expect degraded performance there.
• Not intended as a medical or news fact-checker; the medical and news labels indicate topic, not accuracy or trustworthiness of the content.
• Not tested on long-form documents, articles, or multi-topic paragraphs — designed for short, single-topic snippets (roughly 6–20 words).

Training Data

260 hand-written examples, 65 per class, split across Arabic (MSA + Egyptian colloquial) and English. Each class deliberately includes both formal and casual register examples (e.g. historical_scientific includes both textbook-style facts and "TIL..."-style casual trivia). This was a direct fix after an earlier version of the dataset accidentally let the model learn tone (formal vs. casual) as a shortcut for topic, rather than actual topic.

The dataset is synthetic/hand-written for prototyping purposes, not sourced from real news feeds or social platforms — see Limitations below.

Evaluation

Because SetFit few-shot models can score deceptively well on a small in-distribution eval split, this model was additionally checked against an independent hard test set: 20 examples per class, written separately from the training data, including code-switched sentences, ambiguous phrasing, and registers not seen during training.

Split	Accuracy	Macro F1
Standard eval split (in-distribution)	0.9615	0.9613
Independent hard test set	0.90	0.8958

The in-distribution eval score is still the less trustworthy of the two — with only ~13 examples per class held out, it's a small sample. The hard test set (20 independently-written examples per class, including code-switching and casual/ambiguous phrasing) is the more meaningful number, and this model beat the alternative base models on it.

Per-class breakdown on the hard test set:

Class	Precision	Recall	F1
news	1.00	1.00	1.00
historical_scientific	1.00	0.60	0.75
medical	1.00	1.00	1.00
non_news	0.71	1.00	0.83

Model Comparison

Three base sentence-transformer models were fine-tuned with SetFit and compared on the same independent hard test set; this checkpoint (MiniLM-L12) was selected as the best performer:

Base model	Hard test accuracy	Hard test macro F1
paraphrase-multilingual-MiniLM-L12-v2 (this model)	0.90	0.8958
mDeBERTa-v3-base-mnli-xnli	0.85	0.8352
mDeBERTa-v3-base-xnli-multilingual-nli-2mil7	0.85	0.8520

Known Weaknesses

Training data was deliberately rebalanced to include both formal and casual register in every class (see Training Data above), which meaningfully improved robustness — historical_scientific recall on the hard test set went from 0.20 (pre-rebalancing) to 0.60, and hard-test macro F1 rose from 0.71 to 0.90. That said, two residual weak points remain in this checkpoint:

• historical_scientific recall (0.60) is still the lowest of the four classes — some casually-phrased science/history facts still get misclassified, most often as non_news.
• non_news precision (0.71) is the lowest precision score — it still catches some casual-but-actually-on-topic text (e.g. casual health or trivia posts) as a false positive. Both point the same direction: when text is short, casual, and topically ambiguous, the model still leans toward non_news. If this matters for your use case, consider adding more casual-register historical_scientific examples specifically, or applying a class-specific confidence threshold rather than treating all four classes symmetrically.

Limitations & Bias

• Small, synthetic dataset. All examples were hand-written rather than sourced from real news outlets or social media, so real-world phrasing, slang, spelling variation, and noise (typos, transliteration, mixed scripts) are likely underrepresented.
• Dialect coverage. Egyptian colloquial Arabic only; other dialects are out of distribution.
• Class boundaries can be genuinely ambiguous (e.g. a casually-phrased medical news announcement could reasonably be news or medical) — the model will pick one, but treat low-confidence predictions near class boundaries with caution.
• No fairness/bias audit has been performed on this model; use with appropriate caution in any user-facing or moderation context.

Training Hyperparameters

• batch_size: 32
• num_epochs: 2
• train/eval split: 208 / 52 examples (80/20, stratified across 260 total)
• unique contrastive pairs generated: 32,448
• sampling_strategy: oversampling
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• use_amp: True

Framework Versions

• Python: 3.12.13
• SetFit: 1.1.3
• Sentence Transformers: 5.0.0
• Transformers: 4.53.2
• PyTorch: 2.7.1+cu126
• Datasets: 3.6.0
• Tokenizers: 0.21.4

Citation

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}