SinhalaMMLU-SharedTask/team-mission-x

Team mission_x – Sinhala MMLU Shared Task System

• Team: mission_x (Codabench participant mission_x)
• Members: A.G. Aditha Buwaneka Wimalasuriya (corresponding member), U.H. Chathurya Hewage – University of Moratuwa, Sri Lanka
• Final Codabench submission: ID 148 – Easy 62.7726, Medium 54.3654, Hard 68.4536, Macro 61.8639

System summary

• Base model(s): google/gemma-4-E4B-it, pinned revision ee0ef6023621cff504d758262d4e04895a5af4a2
• What we did: Zero-shot prompting of one model, no training. Every question is first answered by option scoring (next-token log-probability of the option numbers 1-4, averaged over the 4 cyclic option orderings to remove position bias). Every question is then answered in the chat template's thinking mode (greedy decoding, up to 2,048 new tokens, final Answer: N); that answer is used, and the option-scoring answer is the fallback when no answer can be read (89 of 2,238 test questions, mostly reasoning cut off at the token limit). The instruction is a fixed English sentence plus the subject name, identical for every question.
• Total inference parameters: 7,941,100,832 loaded (7.94B; checkpoint total 7,996,156,490), one model, no adapters
• Extra data used for training or retrieval: none. Method choices (prompt, option orderings, thinking vs. scoring) were checked on the released dev set and on the Sinhala part of Global-MMLU (CohereLabs/Global-MMLU, config si); nothing was trained on them.

Models and resources in this submission

Repo	What it is
SinhalaMMLU-SharedTask/team-mission-x	code, inference script, final predictions (this repo)
google/gemma-4-E4B-it	public base model, used unchanged (no fine-tuned model repository)

How to run

pip install -r requirements.txt
python inference.py

• GPU used: Kaggle, 2x NVIDIA T4 16 GB, fp16 (torch 2.11.0+cu128, transformers 5.21.0). The model is spread over all visible GPUs in one process.
• Approximate run time on the test set: ~13.5 h on 2x T4 (table below).
• Anything else the organizers need to know:
  • The run is longer than one 12 h Kaggle session. Each step writes its own file under outputs/ and is skipped when that file exists, so if a session ends, run python inference.py again in the same folder (with outputs/ kept) and it continues from the next step.
  • Decoding is deterministic (no sampling, seed 42). Batch sizes (4 for scoring, 16 for thinking) and the group order in inference.py match the submitted run, since batching changes padding.
  • The script writes predictions.jsonl with the template's section 3 (compact JSON, level from metadata.difficulty). The answers are the same as in the submitted predictions.jsonl, which was written by src/smmlu.py (same ids and order, different JSON spacing). Checked by re-running inference.py on the test dataset with the saved step outputs: 2,238 / 2,238 answers identical.
  • The model is not gated; no token is needed.

Step	Questions	Time (2x T4)
Option scoring (4 orderings)	2,238	~25 min
Thinking, hard	485	~2.9 h
Thinking, medium social science + other	386	~2.3 h
Thinking, easy stem + social science + other	351	~2 h
Thinking, easy humanities	291	~1.7 h
Thinking, medium language	159	~0.9 h
Thinking, medium humanities	566	~3.2 h
Combine	2,238	< 1 min (CPU)

Files

• inference.py – the exact script that produced our final Codabench submission (organizers' template; sections 1 and 3 unchanged)
• src/smmlu.py – model code called by inference.py: run (option scoring), generate (thinking mode), combine (merge)
• predictions.jsonl – the exact file we submitted to the final leaderboard (Codabench submission 148)
• requirements.txt – package versions we used
• manifest.yaml – model, revision, parameter count, decoding, hardware
• pipeline.png – system diagram
• training/ – data-preparation code; no training was done for the submission (see training/README.md)