casperhuggingface/online-chautari-moderator

Online Chautari moderator

A small comment classifier that labels a Nepali or English comment hide or keep. It is the model trained in chapter 4 of the Programiz ML Engineering course, for Online Chautari, a fictional Nepali comment platform. It is a course artefact, not a production moderator.

What it does

Input: one comment in romanised Nepali, Devanagari, English, or a mix. Output: two scores, keep (label 0) and hide (label 1). The course uses a threshold of 0.5 on the hide score.

The labels follow the moderation policy in the dataset card of casperhuggingface/online-chautari-comments. In short: hide attacks on a person, threats and spam; keep criticism of a product, service or performance, however harsh.

How it was trained

• Base model: microsoft/Multilingual-MiniLM-L12-H384 (MIT licence), with a 2-label classification head.
• Data: the dataset's train split, divided 90/10 stratified by label into 5,565 training rows (391 hide) and 618 validation rows. The near-duplicate split was not used.
• LoRA: r 8, alpha 16, dropout 0.1, on the attention query and value modules. 148,226 trainable parameters.
• Optimiser: AdamW, learning rate 5e-4, 8 epochs, batch 16, warmup 10%, weight decay 0.01, max length 64 tokens, fp32, seed 42.
• Validation loss fell from 0.2527 after epoch 1 to 0.1297 after epoch 8. Training took 143.9 s on an Apple M3 (8 GB) with PyTorch on MPS.
• The adapter was merged into the base weights with merge_and_unload.
• Vocabulary trim: the tokenizer and embedding matrix were cut from 250,002 pieces to 7,336, keeping the pieces seen in the training comments plus the special tokens. This took parameters from 117,654,530 to 24,457,346. The trimmed tokenizer matched the original on 500 of 500 training comments. Words outside the kept vocabulary are split into smaller kept pieces; 87 of the 300 held-out comments contain at least one such word.
• Export: ONNX fp32, then int8 with onnxruntime quantize_dynamic (QInt8).

Other learning rates were tried and rejected: 2e-4 for 3 epochs and 1e-3 for 12 epochs both predicted keep for every comment; 1e-3 for 6 epochs reached held-out hide F1 0.529.

Scores

Held-out split: 300 comments, 21 hide. Hide is the positive class.

File	Accuracy	Hide precision	Hide recall	Hide F1	TP / FN / FP / TN
onnx/model.onnx (fp32)	0.970	0.929	0.619	0.743	13 / 8 / 1 / 278
onnx/model_quantized.onnx (int8)	0.967	0.867	0.619	0.722	13 / 8 / 2 / 277

Devanagari stress split (150 comments, 27 hide):

File	Accuracy	Hide recall	TP / FN / FP / TN
fp32	0.867	0.259	7 / 20 / 0 / 123
int8	0.873	0.296	8 / 19 / 0 / 123

Caveats:

• The held-out split has only 21 hide comments, so one comment moves recall by about 5 points.
• The model misses about 4 in 10 hide comments on held-out and about 3 in 4 on Devanagari. Devanagari is 13% of the training data.
• The int8 file is slightly better than fp32 on Devanagari recall by one comment. That is noise at this size, not an effect of quantisation.
• The test data is synthetic, so these scores do not measure performance on real comments.
• For comparison, a LoRA fine-tune of xlm-roberta-base (278M parameters) on the same data reached held-out hide F1 0.762 and Devanagari recall 0.481.

Sizes and speed

File	Size	CPU time per comment
onnx/model.onnx (fp32)	98.0 MB	4.9 ms
onnx/model_quantized.onnx (int8)	25.0 MB	2.5 ms

Times are onnxruntime on an Apple M3 CPU, one comment at a time. transformers.js loads the int8 file when you pass dtype: 'q8'.

Usage with transformers.js

import { pipeline } from '@huggingface/transformers';

const moderator = await pipeline(
  'text-classification',
  'casperhuggingface/online-chautari-moderator',
  { dtype: 'q8' },
);

const scores = await moderator('Timro dimag chaina ki kya ho?', { top_k: null });
// [{ label: 'hide', score: ... }, { label: 'keep', score: ... }]

The ONNX inputs are input_ids and attention_mask (int64), and the output is logits.

Intended use

Teaching: running a small classifier in the browser, comparing fp32 and int8, and reading per-script error rates. It belongs to the Programiz ML Engineering course.

Limitations

• Trained on the dataset text before 17 invented spam web addresses were replaced with -chautari-demo names for publication. The change touches 84 spam strings and no labels.
• Not a production moderator. Do not use it to hide real people's comments.
• Trained only on synthetic, mild, stylised abuse. It has not seen slurs or caste, ethnic, religious or gender-identity terms, which were removed from the data by design.
• Weak on Devanagari, as the scores above show.
• The trimmed vocabulary covers only pieces seen in training. Text far from the training topics is split into many small pieces and is likely to score worse.
• Comments longer than 64 tokens were never seen in training.