SinhalaMMLU-SharedTask/team-nexus

Team Nexus – Sinhala MMLU Shared Task System

Team

Name	Email
S.A. Talagala	samadhianjanatalagala@gmail.com
Anas Hussaindeen	anashussaindeen@gmail.com
Ifadha Imthiyas	ifa.imty015@gmail.com
V.D.W. Muthumala	vihangamuthumala@gmail.com

System summary

• Base model(s): google/gemma-4-E4B-it, commit ee0ef6023621cff504d758262d4e04895a5af4a2, used text-only: the vision and audio towers are removed and every language-model weight is copied unchanged (training/scripts/make_text_only.py checks that the copy is exact).
• What we did: prompting only, with no fine-tuning, retrieval or extra data at inference. Every question is answered in Gemma 4's thinking mode with greedy decoding (no sampling) and up to 3,072 new tokens, and the final choice (1–4) is read from the model's answer. A fast one-step answer (next-token scores over the labels 1–4, averaged over 4 cyclic orders of the options) is computed too, but is used only as a fallback if a thinking answer cannot be read; this happened for 1 of 2,238 test questions.
• Total inference parameters: 7.463B (7,463,013,376; one model, no adapter). This is the count of the weights actually loaded, with tied embeddings counted once.
• Extra data used for training or retrieval: none in the final system. See "Methods and data" below for everything we used during development.

Models and resources in this submission

Repo	What it is
SinhalaMMLU-SharedTask/team-nexus	Code, inference script and final predictions (this repo)
SinhalaMMLU-SharedTask/team-nexus-gemma-4-e4b-it-text	Text-only Gemma 4 E4B-it: unchanged weights of google/gemma-4-E4B-it with the vision and audio towers removed (Apache 2.0). inference.py pins commit 8f686039aad1a865c9335c17874d9b870fdec766.

How to run

pip install -r requirements.txt
python inference.py

inference.py follows the organisers' template. Section 1 loads every .json file in DATA_REPO. Section 2 downloads the model from the repo above and runs our pipeline (inference/predict.py with configs/submission_base_think_greedy.yaml), the code that produced our Codabench file. Section 3 writes predictions.jsonl.

• GPU used: 1× NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified CPU/GPU memory), bf16.
• Approximate run time on the test set: about 4.5 hours (16,277 s for 2,238 questions, including the model download) on the GB10. Most of this is thinking-mode generation; faster GPUs need proportionally less.
• Anything else the organizers need to know:
  • Memory: generation uses batches of up to 64 sequences, with a padded prompt budget of 16,384 tokens and up to 3,072 new tokens each. Plan for a GPU with 80 GB or more, such as an A100 or H100 80 GB. A 16 GB T4 is not enough.
  • Decoding: greedy only (do_sample=False behaviour); there is no sampling anywhere in the submitted system. The seed (1234) is still set.
  • Reproducibility: thinking outputs are long (hundreds to thousands of tokens) and generated in bf16 in batched, length-sorted, left-padded batches. Answers can therefore depend on the GPU type and on which questions share a batch. We ran the published inference.py (this repo, model downloaded from the repo above, fresh environment) on the full test set, and its answers differed from our earlier full run on the same machine in exactly 127 questions, the same number of answers that earlier run's sampling vote had changed. A run on a much smaller subset changes the batches and can differ by a minority of answers, so please run the full dataset with the default settings. On a different GPU type, expect a minority of answers to differ, with a similar overall score.
  • Earlier variant: setting SELF_CONSISTENCY = True at the top of inference.py runs an earlier variant that adds seeded sampled reasoning paths and a majority vote on part of the questions. We do not submit it, because the task asks for no sampling.
  • Gating: the model repo is public and the base model is not gated, so no login is needed.

Files

• inference.py – the entry point that runs the exact pipeline behind our final Codabench submission
• inference/predict.py, eval/ – the pipeline code it calls (prompting, scoring, thinking, self-consistency)
• configs/submission_base_think_greedy.yaml – the settings of the submitted system; configs/submission_base_think_gated_sc.yaml – the earlier sampling variant (not submitted)
• predictions.jsonl – the exact file we submitted to the final leaderboard (sha256 b28acab09e6c926b3f40b487af393937c9d19eafd98445a83dec48552852facf)
• requirements.txt – package versions we used (Python 3.12.3)
• training/ – all our data-preparation, training and evaluation code and configurations from development (see below)

Methods and data (complete list)

The final system uses no training data and no fine-tuning: it is the base model, prompted, with greedy thinking (see "System summary"). The only data it touches are the test questions it is asked to answer. Everything below is development work in training/; none of it is part of the final system, and none of it beat the base model with thinking on our held-out data.

Methods we tried

1. Prompting: English zero-shot (our baseline), Sinhala-language instructions, fixed few-shot, and other prompt variants. None beat plain English zero-shot on Gemma 4 E4B.
2. Option-order averaging: averaging the label probabilities over 4 cyclic orders of the options. Used for the fast fallback answer; also tested as option-order voting over thinking runs (rejected by our pre-registered decision rule).
3. Model comparison and parameter accounting: screening of open ≤ 8B candidates (Gemma 4 E4B/E2B, Gemma 3n E4B/E2B, Gemma 3 4B, Qwen3 and Qwen3.5 4B, Llama 3.2 3B, SmolLM3 3B, Phi-4-mini, Granite 4.x 3B), plus exact parameter counts (training/MODEL_MANIFEST.md). Gemma 4 E4B-it was the strongest of those we evaluated.
4. Thinking mode: the model's built-in reasoning before the answer. This is the main gain and is used in the final system.
5. LoRA fine-tuning (SFT): on Global-MMLU si questions, and on SinhalaMMLU dev with 5-fold cross-validation (never scored on its own training questions).
6. Continued pre-training (LoRA): on cleaned Sinhala Wikipedia, FineWeb-2 and NSINA, with about 10% English FineWeb-Edu.
7. Distillation from larger open teachers: soft labels on Global-MMLU si training questions from locally run open models (Gemma 4 31B-it for the training run; Qwen3.5-27B and Gemma 4 26B-A4B-it were also evaluated as teachers).
8. Synthetic MCQs: a small pilot of Sinhala exam-style questions written by Gemma 4 31B from Sinhala Wikipedia passages, then validated, round-trip checked and soft-labelled by the same model.
9. Reasoning-trace fine-tuning (STaR-style rejection sampling): the base model's own correct thinking traces on held-out SinhalaMMLU dev questions (round 1), plus Global-MMLU si Humanities / Social Sciences / Other / Business questions and the synthetic Wikipedia MCQs (round 2); LoRA with loss on the completion only.
10. Self-consistency: sampled reasoning paths with majority vote, including an agreement-gated variant. Not part of the final system, because the task asks for no sampling. It is still in the code behind SELF_CONSISTENCY = True.

Each step was decided on SinhalaMMLU dev or Global-MMLU validation subsets, never on test questions. All models were run locally and are open-weight. No closed or paid API was used at any stage.

Data sources

Data	Source and revision	Licence	How it was used
Shared-task test questions	SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset, rev 56fb2da50c	set by organisers	Prediction only. Never used for training, tuning, prompt design or model selection
SinhalaMMLU public Easy tier (dev)	naist-nlp/SinhalaMMLU, TEST/easy, rev 7dd30681d9	CC BY-NC-ND 4.0 (not redistributed)	Development evaluation; SFT cross-validation; source of questions for round-1 reasoning traces
Global-MMLU si	CohereLabs/Global-MMLU, rev 0e619dbeb3	Apache-2.0	2,000-question validation sample; remaining questions for SFT, distillation and round-2 traces
Belebele sin_Sinh	facebook/belebele, rev 7899cdfa4e	CC BY-SA 4.0	Evaluation only
Sinhala Wikipedia	wikimedia/wikipedia 20231101.si, rev b04c8d1ceb	CC BY-SA 3.0 / GFDL	Continued pre-training; passages for synthetic MCQs
FineWeb-2 sin_Sinh	HuggingFaceFW/fineweb-2, rev af9c13333e	ODC-By 1.0	Continued pre-training; passages for synthetic MCQs
NSINA news	sinhala-nlp/NSINA, rev a986af81ad	CC BY-SA 4.0	Continued pre-training; passages for synthetic MCQs
FineWeb-Edu (English sample)	HuggingFaceFW/fineweb-edu sample/10BT, rev 87f09149ef	ODC-By 1.0	About 10% of the continued pre-training mix
Synthetic MCQs and reasoning traces	Generated by us from the sources above with local open models	n/a	Training experiments only; not published

Contamination controls (details in training/DATA_LOG.md): web documents from exam and past-paper sites were removed from all pre-training corpora; training questions were filtered by character n-gram similarity against the SinhalaMMLU dev and Global-MMLU validation questions; Global-MMLU was checked for overlap with SinhalaMMLU using character n-gram and multilingual-embedding similarity.

Models used during development (all open-weight, run locally)

Base: google/gemma-4-E4B-it (Apache-2.0). Teachers and comparison models: google/gemma-4-31B-it, google/gemma-4-26B-A4B-it, Qwen/Qwen3.5-27B, and the ≤ 8B candidates listed above. For the overlap checks only: intfloat/multilingual-e5-large (embedding model, MIT licence).

Citation

@inproceedings{pramodya-etal-2025-sinhalammlu,
  title     = "{S}inhala{MMLU}: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in {S}inhala",
  author    = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and
               Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro",
  booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
  year      = "2025",
  pages     = "32931--32949",
  url       = "https://aclanthology.org/2025.emnlp-main.1673/"
}