SinhalaMMLU-SharedTask/team-sinhalanuwana

Team SinhalaNuwana - Sinhala MMLU Shared Task System

Codabench macro score 63.5755 (Easy 64.3302, Medium 56.7057, Hard 69.6907).

System summary

• Base model(s): google/gemma-4-E4B-it, revision ee0ef6023621cff504d758262d4e04895a5af4a2, text decoder only
• What we did: LoRA fine-tuning with STaR (self-taught reasoner): the base model thought about 8,894 Sinhala training questions, and we trained a LoRA adapter on its 4,073 correct first-try thoughts. At inference the model thinks in Sinhala before it answers (thinking mode, greedy). Each question is answered with the choices in several orders, and the option probabilities are averaged. Easy and Hard questions also get a longer thought of up to 2,048 tokens, and Medium questions 4 more choice orders. One model, no retrieval.
• Total inference parameters: 7.50B (7,501,679,616: 7,463,013,376 base and 38,666,240 adapter)
• Extra data used for training or retrieval: Our own Sinhala multiple-choice questions, generated by Claude models from Sri Lankan school textbooks and teacher guides. They are in training/data/ of this repository. No retrieval.

Models and resources in this submission

Repo	What it is
SinhalaMMLU-SharedTask/team-sinhalanuwana	code, inference script, final predictions, training code and training data (this repo)
SinhalaMMLU-SharedTask/team-sinhalanuwana-gemma4-e4b-star3-lora	LoRA adapter on gemma-4-E4B-it: weights, config, tokenizer files, training log
google/gemma-4-E4B-it	public base model (Apache 2.0)

How to run

pip install -r requirements.txt
python inference.py

• GPU used: Kaggle T4 x2 (16 GB each). One GPU with 24 GB or more also holds the model.
• Approximate run time on the test set: about 16 hours for the 2,238 questions on T4 x2 (six passes of 4.3, 4.2, 1.7, 1.9, 2.1 and 2.1 hours)
• Anything else the organizers need to know:
  • Python 3.13. If torchao is installed (Kaggle has it), run pip uninstall -y torchao first, because peft imports it and rejects that version.
  • inference.py calls predict.py and merge_votes.py from the same folder, so run it inside a clone of this repository.
  • It writes predictions.jsonl in the current folder, which replaces our submitted copy in a clone. The submitted file stays in the repository history.
  • Only DATA_REPO needs to change. No token is needed if the repositories are public.

Files

• inference.py - the exact system that produced our final Codabench submission
• predict.py, merge_votes.py - one answering pass and the vote over the passes, called by inference.py
• predictions.jsonl - the exact file we submitted to the final leaderboard
• requirements.txt - package versions we used
• manifest.yaml, model_manifest.json - models, adapter SHA-256, parameter count, closed-API use
• training/ - training and data-preparation code, Kaggle notebooks, settings and training data (see training/README.md)

Team details

Name	Affiliation	Role
Senali Dilumika	UCSC	Corresponding member

Team name on Codabench: SinhalaNuwana.

Verification

• Fresh Kaggle session (2 x T4, 28 October 2026), the same code, the same pinned versions and the same adapter, on the first 3 questions of each of the 41 test files (123 questions): 122 answers are the same as in predictions.jsonl. All Medium and Hard answers are the same. The one changed answer is an Easy question whose 5 votes split 2-2-1. Float16 GPU arithmetic is not fully deterministic, so a full rerun can change a few close votes.
• merge_votes.py, run on the six scored passes, rebuilds predictions.jsonl byte for byte.

Training (details in training/README.md)

training/README.md gives the full pipeline, the settings and the data. In short:

1. Claude models generated Sinhala multiple-choice questions from Sri Lankan school textbooks, NIE teacher guides and resource books, A/L learning outcome notes and Wikibooks. A stronger Claude model checked the answer keys of 10,000 of them.
2. review_overlap.py removed the 1,106 questions that resemble a dev or test question by wording or by meaning (LaBSE cosine 0.70 or more). 8,894 questions remained.
3. star.py let the base model (no adapter) think about each question for up to 512 tokens. It kept the 4,073 thoughts that gave the correct answer on the first try.
4. train_lora.py trained a new LoRA adapter on those thoughts: 2 epochs, AdamW, learning rate 1e-4 with a cosine schedule and 3% warmup, 16 examples per step, sequences of up to 1,024 tokens. It took 4.3 hours on 2 x T4.

Licence and citation

The code and the adapter are released under the Apache 2.0 licence, the licence of Gemma 4.