Jonandrop/camelbert-lemma-ar

Arabic Lemmatizer (int8 ONNX)

Per-language lemmatizer and UPOS tagger fine-tuned from CAMeL-Lab/bert-base-arabic-camelbert-msa on UD Arabic-PADT. Predicts an edit-tree label per token; applying it to the surface word yields the lemma. Dynamic-int8 quantized ONNX (~112 MB).

Accuracy (UD test, dynamic int8, valid-label masking ON)

Metric	int8	fp (reference)
Lemma accuracy	43.91%	43.91%
UPOS accuracy	68.32%	N/A

int8 trades accuracy for size; the fp16/torch model is several points higher (UPOS head is most affected by quantization).

CRITICAL: valid-label masking

Plain argmax over edit-tree labels often picks a label that is structurally invalid for the word (its delete-segment does not match), applying it fails and accuracy collapses. At inference you MUST iterate candidate labels in descending logit order and pick the first whose edit applies to the word (identity is the guaranteed fallback). This is model-internal selection, not a hand-written rule.

Files

• model.int8.onnx: inputs input_ids, attention_mask; outputs upos_logits, lemma_logits
• config.json, tokenizer.json, tokenizer_config.json, special_tokens_map.json
• id2label.json / label2id.json: edit-tree lemma labels
• upos_id2label.json / upos_label2id.json: UPOS labels
• edit_trees.json: edit-tree label definitions
• lexicon.json: fallback {word: lemma} lexicon