SinhalaMMLU-SharedTask/team-qbits

Team QBitsLK – Sinhala MMLU Shared Task System

System summary

• Base model(s): google/gemma-4-E4B (text-only), with Sinhala continued pretraining, then LoRA supervised fine-tuning on examination-paper MCQs.

• What we did: We built a 7,872-question Sinhala multiple-choice dataset by extracting from publicly available Sri Lankan O/L and Grade 6–11 past papers, cleaning OCR artefacts and decontaminating against the released SinhalaMMLU test split. We continued pretraining Gemma 4 E4B on 1M sentences from the public Minuri/diverse_sinhala_dataset corpus (LoRA r=128, alpha=32, rsLoRA, packed same-domain pseudo-documents, 1 epoch), then fine-tuned that checkpoint with LoRA (r=32, alpha=32, 2 epochs) using answer-only loss, and merged the adapter back into the weights. At inference each option is scored by per-option log-likelihood under a 5-prompt ensemble (Sinhala and English instructions, subject intro on/off, two option-order rotations) rather than by generation. Nothing is sampled.

• Total inference parameters: 7,996,157,418 (one merged model; ≤ 8B)

• Extra data used for training:

  1. Minuri/diverse_sinhala_dataset (file high_confidence_7_7M.csv) - public Sinhala text corpus, used for the continued-pretraining stage. https://huggingface.co/datasets/Minuri/diverse_sinhala_dataset
  2. SinhalaMMLU-SharedTask/team-qbits-mcq-data - our own past-paper MCQ dataset (7,872 questions), used for the supervised fine-tuning stage.

  No retrieval, no external APIs, no closed models.

Models and resources in this submission

Repo	What it is
SinhalaMMLU-SharedTask/team-qbits	code, inference script, final predictions (this repo)
SinhalaMMLU-SharedTask/team-qbits-cpt-v3	the submitted model — merged 16-bit, 7,996,157,418 params
QBitsLK/gemma4-e4b-sinhala-cpt	the continued-pretrained base we fine-tuned (Gemma 4 E4B + Sinhala CPT), merged 16-bit
SinhalaMMLU-SharedTask/team-qbits-cpt-v3-lora	the LoRA adapter for the above (r=32), for inspection
SinhalaMMLU-SharedTask/team-qbits-base-v3b	ablation — same recipe on the plain Gemma 4 E4B base, merged
SinhalaMMLU-SharedTask/team-qbits-base-v3b-lora	the LoRA adapter for the ablation
SinhalaMMLU-SharedTask/team-qbits-mcq-data	the SFT dataset (prompt/completion files)

inference.py uses only team-qbits-cpt-v3. The other repos are provided for transparency and are not part of the inference pipeline.

How to run

pip install -r requirements.txt
python inference.py

• GPU used: 1x NVIDIA A40 48 GB (any GPU with ≥ 20 GB should work; the model is 16 GB in bf16)
• Approximate run time on the test set: ~18 minutes for 2,238 questions (5 prompt variants × 4 options = ~44,800 scored candidates); peak VRAM ~27 GB
• Anything else the organizers need to know:
  • The Gemma license must be accepted on the Hugging Face account used to run this. Our merged model is self-contained, so no separate base-model download is required.
  • inference.py scores options by log-likelihood and never calls generate, so there is no sampling. With the fixed seed, repeated runs produce byte-identical predictions.jsonl.
  • VARIANTS at the top of inference.py defines the prompt ensemble. Changing that list changes the predictions; it is set exactly as used for our final Codabench submission.
  • The reference util_prompt.py hard-codes the hard instruction as "1, 2, 3, 4, 5", which was correct for the original five-option hard split. SinhalaMMLU 2.0 reduced hard questions to four options, so fit_numbers() derives the number list from the actual option count. Verified: 0 of 2,238 prompts disagree with their option count.

Results (Codabench)

system	easy	medium	hard	macro
submitted — team-qbits-cpt-v3, 5-prompt ensemble	58.10	62.56	52.79	57.81
the same model with the adapter applied at load time instead of merged	58.26	62.02	52.58	57.62
ablation — team-qbits-base-v3b (plain Gemma 4 base), 5-prompt ensemble	57.01	63.10	51.34	57.15
ablation — plain base, single prompt, no ensemble	—	—	—	55.66

inference.py reproduces the submitted row exactly: it loads the merged model, which is the system we ship. Row 2 is the same fine-tune evaluated with the LoRA applied at load time (base + adapter). That form is 8,069,557,738 parameters and therefore exceeds the 8B cap, so it is not the submitted system; it is listed only because merging is mathematically equivalent yet not bit-identical in bf16 (W + BA rounded once, versus Wx + B(Ax)), which resolves about 1% of near-tied questions differently.

Files

• inference.py – the exact script that produced our final Codabench submission
• predictions.jsonl – the exact file we submitted to the final leaderboard (macro 57.81), byte-for-byte reproducible by running inference.py
• predictions_ablation_base_v3b.jsonl – an earlier submission from the ablation model (team-qbits-base-v3b, macro 57.15), included for transparency. To regenerate it, set MODEL_ID to SinhalaMMLU-SharedTask/team-qbits-base-v3b and change nothing else.
• requirements.txt – package versions we used
• training/ – training, fine-tuning and data-preparation code
  • cpt_gemma4_sinhala.py – stage 1: continued pretraining on Minuri/diverse_sinhala_dataset
  • eval_cpt_success.py – stage 1 diagnostics (held-out bits-per-character, English regression)
  • sft_gemma4_mcq.py – stage 2: LoRA SFT with answer-only loss and candidate-scoring evaluation
  • mcq_dataset.py – extraction cleanup, validation, decontamination, splitting, prompt building
  • add_new_data.py, normalize_subjects.py – dataset assembly and subject-name alignment
  • merge_adapter.py, push_merged_to_hf.py – adapter merging and publishing
  • eval_sinhalammlu.py – read-only analysis; also builds the decontamination blocklist
  • check_env.py – dependency verification
  • run_data_pipeline.sh, run_training.sh, run_inference.sh – end-to-end drivers
  • REPRODUCE.md – exact commands, expected counts and the training curve

Training pipeline (two stages)

Stage 1 - continued pretraining. training/cpt_gemma4_sinhala.py on Minuri/diverse_sinhala_dataset (high_confidence_7_7M.csv)[Minuri/diverse_sinhala_dataset], 1M sentences, domain-balanced sampling, 16 same-domain sentences joined into each pseudo-document, packing enabled, LoRA r=128 alpha=32 with rsLoRA, 1 epoch on one A40. Result: QBitsLK/gemma4-e4b-sinhala-cpt (merged). Diagnostics in training/eval_cpt_success.py - held-out bits-per-character improved 19.6% while English perplexity regressed 21.7%.

python training/cpt_gemma4_sinhala.py --max_sentences 1000000 --epochs 1

Stage 2 - supervised fine-tuning. training/sft_gemma4_mcq.py on our MCQ dataset, LoRA r=32 alpha=32, lr 1e-4, 2 epochs, batch 8 x accumulation 4, answer-only loss; checkpoint final_adapter (step 1328), merged into the stage-1 weights to give the submitted model.

bash training/run_training.sh

Note the CPT stage used an r=128 adapter and the SFT stage r=32; both were merged, and merging adds no parameters, so the submitted model is 7,996,157,418 - the same count as the Gemma 4 E4B lineage.

Data and decontamination

Source: publicly available Sri Lankan examination papers (pastpapers.wiki, e-thaksalawa.moe.gov.lk), O/L and Grades 6–11. We do not redistribute the source PDFs; the derived question set is published at team-qbits-mcq-data.

The released TEST/easy split was used only as a decontamination blocklist. It matched and removed 26 questions from our training data. It was never used for training, checkpoint selection, prompt design or hyperparameter choice. Model selection used a held-out split of our own data, grouped so that no source paper appears in both training and validation.

Citation

This work evaluates on the SinhalaMMLU benchmark:

Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, and Taro Watanabe. 2025. SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32943–32961. https://aclanthology.org/2025.emnlp-main.1673/