SinhalaMMLU-SharedTask/team-rexx

Team Rexx: Sinhala MMLU Shared Task System

Team members

Name	Affiliation
P.L.M.S. Udayashan	University of Sri Jayewardenepura
K. Praveen	University of Ruhuna

Final submission: Codabench submission ID 145 by account kanishkapg (our submission 09), on 28 October 2026 at 19:13: greedy decoding, made by inference.py in this repo. It scored Easy 70.7165, Medium 59.4059, Hard 72.5773, macro average 67.5666 (first on the leaderboard when we uploaded this repo).

Earlier submission: Codabench submission ID 104 (our submission 08), on 27 October 2026 at 20:19, made with the same model and recipe code but sampled decoding (16 answers at temperature 1.0). It scored Easy 66.5109, Medium 57.6058, Hard 69.6907, macro average 64.6025. We replaced it with the greedy run so that the leaderboard file comes from a deterministic script, as the template asks. Its files are kept in submission08/.

System summary

• Base model: google/gemma-4-E4B-it. It is open weights, created on the Hub on 2 March 2026, and licensed Apache 2.0 under the Gemma 4 licence.
• What we did:
  • Distillation. We fine-tuned the base model with LoRA on 20,870 English reasoning traces for Sinhala curriculum MCQs. Only traces whose answer matched the key were kept.
  • Training questions. They come from three sources: questions written from official Sinhala-medium textbooks and teacher guides, the official development set, and blind-checked machine-written questions.
  • Merge and averaging. We merged the adapter and averaged the result 50/50 with the base weights (WiSE-FT).
  • Inference. Each question gets 8 greedy reasoning answers, each with the options in a different order. The majority answer wins.
• Total inference parameters: 7,941,100,832 (7.94 B), in one model.
  • It is BF16 and not quantised. No adapter, ensemble, router, classifier or retriever is loaded.
  • The count covers every tensor in the checkpoint, including the vision and audio towers that text-only inference never uses.
• Extra data used for training: listed under Training data. Nothing is used for retrieval.

Models and resources in this submission

Repo	What it is
SinhalaMMLU-SharedTask/team-rexx	Code, inference script, final predictions, training code (this repo)
SinhalaMMLU-SharedTask/team-rexx-gemma-4-e4b commit f6936800	The fine-tuned model: full merged BF16 weights, tokenizer, chat template and configuration. inference.py loads this commit.
SinhalaMMLU-SharedTask/team-rexx-training-data (dataset)	The 18,635 machine-written training rows, the model behind each closed-model row, and the list of the 930 official development questions we trained on (file and q_no only, since that dataset is gated)
sinhala-mmlu/gemma-4-e4b-sinhala-mmlu tag v2.0.0	Our public release of the same weights, with the full model card and benchmark. All its files except README.md have the same SHA-256 in the organisation copy.
SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset revision 56fb2da5	The test release our submission answered

How to run

pip install -r requirements.txt
python inference.py

• GPU used: one NVIDIA A100-SXM4-40GB, on Google Cloud Vertex AI in Google's Colab runtime image (Python 3.13), as a fresh environment. The script needs one GPU with at least 40 GB, such as an A100 or H100; we have not tested smaller GPUs.

• Run time on the test set, measured on 28 October 2026 from a fresh environment on one A100 40 GB:

  Step	Time
  pip install -r requirements.txt	2 min 41 s
  python inference.py: downloading the 16 GB of weights and starting vLLM (compilation, CUDA graphs)	7 min
  python inference.py: answering 2,238 questions × 8 answers	49 min
  Total	59 min

• Outputs:

  • predictions.jsonl in the Codabench format.
  • predictions_details.jsonl with the 8 votes and the per-answer results (option order, parsed answer, token count) for each question.

• Anything else the organizers need to know:

  • No login needed. The model and the test set are public, and no gated model is used.
  • The script removes two packages. Before loading vLLM it uninstalls torchaudio and torchcodec, if they are installed. The torchaudio build that vllm==0.32.0 brings from PyPI targets another CUDA version than PyTorch 2.13.0 and stops Transformers from importing, and this text-only system uses neither package.
  • Determinism.
    • Decoding is greedy (temperature 0, the vLLM equivalent of do_sample=False), the seed is fixed, and VLLM_ENABLE_V1_MULTIPROCESSING=0 makes vLLM's scheduling deterministic.
    • On one machine, re-runs are identical. tests/test_determinism.py runs the system twice, each in a fresh process with its own vLLM engine, on every 19th test question. All 944 answers (118 questions × 8) were identical in parsed answer, token count, finish reason and fallback digit, and so were all 118 final answers.
    • On another machine, about 92% of the final answers are reproduced. On 28 October 2026 we downloaded this repo onto a second A100-SXM4-40GB machine with the same container image and package versions, then ran pip install -r requirements.txt and python inference.py. The new predictions.jsonl agreed with ours on 2,050 of 2,238 questions: 91.6% overall, Easy 93.9%, Medium 89.6%, Hard 93.0%. final_run/rerun_check/ has the log and the comparison.
    • Why they differ. Each machine compiles and tunes its own GPU kernels, so the rounding differs slightly. Over long reasoning (876 tokens per answer on average) these differences change the text of most answers and the final answer of a few questions. vLLM's batch-invariant mode (VLLM_BATCH_INVARIANT=1) did not close the gap: two machines agreed on 106 of 118 final answers.
    • What to expect. A re-run on another machine will not reproduce predictions.jsonl byte for byte, but should score close to it. On our validation sets, re-runs of the same model on other machines (A100, L4) changed the mean accuracy by less than half a point.
  • Model download. Our final run downloaded the weights from sinhala-mmlu/gemma-4-e4b-sinhala-mmlu at tag v2.0.0. inference.py now loads the organisation copy SinhalaMMLU-SharedTask/team-rexx-gemma-4-e4b, whose 26 files other than README.md have the same SHA-256 as that tag; this is the only change to the script's code since the run.
  • Checking a re-run. inference.py overwrites predictions.jsonl in its folder. python tests/compare_predictions.py predictions.jsonl compares the new file with the submitted one, which it downloads from this repo; final_run/predictions_details.jsonl has our per-answer results for a closer comparison.

How inference works

inference.py follows the organisers' template, and its section 2 copies the recipe functions verbatim from training/bakeoff_run.py, the script that produced submission 08. Only the decoding settings differ from submission 08: GREEDY = True and SAMPLES = 8, where submission 08 sampled 16 answers at temperature 1.0. GREEDY sets the sampling parameters to temperature 0, exactly what that script's --greedy option does.

1. Load every .json file of DATA_REPO with the template's loader. Order the questions as in submission 08: levels easy, medium, hard, then files by name. Split them round-robin into 4 parts, as submission 08 did.
2. Build the prompt. A fixed English system prompt sets the task ("expert teacher of the Sri Lankan national school curriculum"). The user turn gives the subject and grade from the question's metadata, the Sinhala question and the four numbered options. The Gemma chat template is used with thinking on.
3. Generate 8 greedy answers per question, up to 7,168 tokens each, with a context of 8,192 tokens.
   • Answer k shows the options in order k of 8. The orders are the 4 rotations of a fixed per-question order (seeded by the CRC32 of the question id) and their reverses.
   • Options that refer to other options ("above", "1 and 2") keep their order.
4. Read the last Answer: N after the thinking and map it back to the original numbering. If an answer has no Answer: N, close its thinking and take the most probable digit 1–4. This happened for 15 of the 17,904 test answers.
5. Take the majority vote over the 8 answers; a tie goes to the higher mean option probability.

Why 8 answers over shuffled options. On our validation sets, 600 locked official development questions (easy) and 1,000 Global-MMLU Sinhala questions (a proxy for hard), the vote scored 69.00 and 77.80. That is 1.0 point above one greedy answer in the original option order (68.00 and 76.80, the model card's benchmark run). In the same run as the vote, one greedy answer in a shuffled order scored 65.50 and 75.30 on average; against that, the vote adds 3.5 points (95% CI +1.9 to +5.1) and 2.5 points (+1.5 to +3.5). Setting SAMPLES = 1 gives one greedy answer per question in about 17 minutes. On the test set, our final predictions agree with submission 08 on 85.6% of questions.

There is no retrieval, knowledge base, internet access at inference, tool use or closed API, and no human in the loop.

Checks, all passed on 28 October 2026:

• python tests/test_recipe_identical.py (no GPU): the recipe code is unchanged from submission 08, and every argument except the decoding settings matches the 4 original job reports in submission08/run_reports/.
• python tests/test_inputs_identical.py (no GPU): the loader rebuilds exactly the inputs of submission 08, with the same SHA-256 as logged by the original jobs.
• python tests/test_determinism.py (GPU): two fresh runs on one machine give identical answers, as above.
• python tests/compare_predictions.py <file>: compares any re-run with our submitted predictions.jsonl. --submitted submission08/predictions.zip compares with submission 08 instead.

Training data

All training rows are Sinhala MCQs with an English reasoning trace and the keyed answer (training/README.md gives the full lineage and commands).

Source of the questions	Rows
Written by Gemma 4 26B-A4B (open) from passages of the official Sinhala-medium textbooks (grades 6–11, Educational Publications Department) and the A/L teacher guides and resource books (NIE, e-thaksalawa)	14,571
Written by closed models from the same kind of passages, kept only if a second model answered them correctly without the key	4,064
Official development set naist-nlp/SinhalaMMLU: 930 questions outside the 600 we kept locked for validation	2,235

Source of the reasoning traces	Traces
Gemma 4 26B-A4B-it and 31B-it (open, run offline on our GPUs)	11,334
Closed models, answering blind with tools and web search off	9,536

Closed models, used only to generate training data before the test phase ended, never at inference:

Use	Model	Rows
Write MCQs from textbook passages	OpenAI Codex CLI gpt-6.1-sol	3,968
Write MCQs (pilot round)	Anthropic Claude, via Claude Code	96
Blind-check generated MCQs	Anthropic Claude Sonnet (Claude Code) and OpenAI Codex gpt-6.1-sol	filter only
Reasoning traces	OpenAI Codex CLI gpt-6.1-sol	9,027
Reasoning traces	xAI Grok CLI grok-4.7-build-fast	397
Reasoning traces	Anthropic Claude Code claude-opus-5-5	112

A closed trace was kept only if all of these held:

• its answer matched the key, and its reasoning ended on the same option;
• it had 120–900 words;
• it did not mention a passage, an answer key or a URL.

An early Codex batch could reach a server-side search feature. We switched that feature off, and the URL filter removed every trace citing a web page.

Test data:

• No test question, option or answer is in the training data. training/overlap.py finds 0 exact question+options matches; the only shared stems are generic ones from the development set.
• The prompt and recipe were chosen on development questions and on Global-MMLU Sinhala.
• The test release's subject and level counts, and the per-subject vote agreement of our earlier test predictions, guided which curriculum subjects received more generated training questions.
• We used no past exam papers.

Files

• inference.py: the script that produced our final Codabench submission (its recipe is a verbatim copy of training/bakeoff_run.py)
• predictions.jsonl: the exact file we submitted to the final leaderboard (SHA-256 86f5d0c88881763a17097ec33a3457e143f7a7e049ce2b604a5730fc57f090e4), uploaded to Codabench zipped on its own as predictions.zip
• requirements.txt: package versions used
• final_run/: the logs of the fresh-environment run that wrote predictions.jsonl: job log, inference log, determinism-test log, pip freeze, and predictions_details.jsonl. final_run/rerun_check/ holds the job log and comparison of the re-run on a second machine.
• training/: training, fine-tuning and data-preparation code and configurations, with the Vertex AI job specs as run
• tests/: the checks above
• submission08/: our earlier sampled submission: the uploaded predictions.zip and the reports of its 4 inference jobs (arguments, vLLM version, parse and thinking-closure rates)

Citation

This system was built for the SinhalaMMLU shared task. It uses the SinhalaMMLU benchmark:

Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito and Taro Watanabe. 2025. SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32943–32961.