coderGit/eng-pidginedu-toucan

PidginEdu-LLM (Eng-PidginEdu Toucan-1.2B): English to Nigerian Pidgin, glossary-augmented

Full fine-tune of UBC-NLP/toucan-1.2B on the Eng-PidginEdu glossary-augmented English to Nigerian Pidgin dataset (26,232 sentence pairs, 8 academic subjects). Produces inline terminology glosses alongside the translation, e.g. an English term followed by a parenthetical Pidgin explanation.

This is the finalized PidginEdu-LLM flagship model from the Eng-PidginEdu benchmark. Full methodology, every hyperparameter, every bug found and fixed, and the full 14-model comparison are documented in the benchmark's GitHub repository: shashacode/Eng-PidginEdu-Benchmark.

Why this model, and why not the automated-metric leader

14 models were compared under full fine-tuning, zero-shot, and LoRA/PEFT against four automated metrics (BLEU, chrF++, AfriCOMET, glossary-accuracy). By that automated process, a different model (mt5_large) actually leads 3 of 4 metrics among the 12 fully fine-tuned models -- not this one.

PidginEdu-LLM's flagship designation was decided differently: after the automated process completed and was independently validated (a validation-set leaderboard confirmed the same automated ranking as the test set, closing a methodological gap where the original comparison had picked its winner using test-set scores), the project author reviewed generated translations from both models directly and judged this model's output to read as more natural, more fluent Nigerian Pidgin -- despite its lower scores on every automated metric used in the benchmark.

This is disclosed here as exactly what it is: a qualitative, single-reviewer judgment call, not a new evaluation protocol. The benchmark's own report documents this transparently, including the automated-metric leader's full results, rather than presenting this model's selection as if it were the output of the metrics-only process -- see the benchmark report, section 9.5, for the complete reasoning on both sides.

Results (test set, beam-5 decoding)

Metric	Score
BLEU	64.74
chrF++	79.34
AfriCOMET	71.80
Glossary accuracy	75.35
Glossary F1	73.96

Training

• Base checkpoint: UBC-NLP/toucan-1.2B (1.2B parameters)
• Full fine-tuning (all parameters), fp32
• Optimizer: Adafactor, learning rate 1e-4 (corrected from an initial 1e-3 attempt that failed to converge -- see the benchmark report section 3.17 for the full incident)
• Batch size 2/GPU, effective batch 16, 5 epochs, 2x Tesla V100-32GB
• Target: pcm_augmented (the glossary-augmented translation)
• Prefix: "translate English to Pidgin: " (the original "<2pcm> " language-tag prefix this checkpoint's upstream card suggests is not a registered vocabulary token and fragments into meaningless pieces -- see the benchmark report section 3.17)

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("coderGit/eng-pidginedu-toucan")
model = AutoModelForSeq2SeqLM.from_pretrained("coderGit/eng-pidginedu-toucan")

text = "translate English to Pidgin: " + "The delegates wanted George Washington to be president."
inputs = tokenizer(text, return_tensors="pt")

# This tokenizer emits token_type_ids, which this model's generate() does
# not accept -- filter to just input_ids/attention_mask first.
gen_inputs = {k: v for k, v in inputs.items() if k in ("input_ids", "attention_mask")}

output = model.generate(**gen_inputs, num_beams=5, max_length=200)
print(tokenizer.decode(output[0], skip_special_tokens=True))

License

Non-commercial use only. This fine-tune is a derivative of UBC-NLP/toucan-1.2B, which is released under a gated agreement restricted to non-commercial use, requiring citation of the original Cheetah and Toucan papers. Those terms apply to this derivative model too. See UBC-NLP/toucan-1.2B for the original terms and required citations before using this model.

Citation

If you use this model, please cite both the original Toucan/Cheetah papers (per the upstream license terms) and the Eng-PidginEdu benchmark: shashacode/Eng-PidginEdu-Benchmark.