SinhalaMMLU-SharedTask/team-hakuru-labs

Team hakuru labs – Sinhala MMLU Shared Task System

Team members (individual participation): Tharini Anjalee Fernando

System summary

• Base model(s): iCIIT/redqueenprotocol-sin-llama3.2-3B-model & Jainamshahhh/helalm-4b
• What we did: We fine-tuned two LoRA adapters (one on Llama-3.2-3B and one on Gemma-3-4B) using QLoRA. Our approach was inspired by knowledge distillation and small language model reasoning techniques. At inference, we use a difficulty-aware router: Easy/Medium questions are answered by an ensemble of both models, while Hard questions are answered exclusively by HelaLM. We use 5-run self-consistency and human heuristics (absolute word penalty and lexical overlap) to select the final answer.
• Total inference parameters: 7.24B (3.22B + 4.02B), safely under the 8B limit.
• Extra data used for training or retrieval:
  • Synthetic Data: ~24,500 Sinhala MCQs generated using closed APIs (DeepSeek, Claude, and ChatGPT). The generation prompts and scripts are provided in the training/ folder. A 50-question sample is provided in training/data_sample.jsonl.
  • Full Dataset: The separate full training dataset and validation set can be accessed here: Google Drive Link It is included in this repo under "data/"
  • Source Material: Publicly available Sri Lankan government school textbooks and teacher guides (Grades 6–13) downloaded from GovDoc.lk (https://govdoc.lk/). A full breakdown of the exact textbooks and grades used is provided in training/data_sources.txt.
  • Public Data: The official SinhalaMMLU Easy-tier development set (naist-nlp/SinhalaMMLU).
  • Training Note: HelaLM was trained on ~13,500 questions, while the RedQueen adapter (v7) was trained on the full 24,561 questions.

Models and resources in this submission

Repo	What it is
SinhalaMMLU-SharedTask/team-hakuru-labs	code, inference script, final predictions (this repo)
SinhalaMMLU-SharedTask/team-hakuru-labs-redqueen	LoRA adapter on RedQueen Llama 3.2 3B
SinhalaMMLU-SharedTask/team-hakuru-labs-helalm	LoRA adapter on HelaLM Gemma 3 4B

How to run

pip install -r requirements.txt
python inference.py

GPU used: Kaggle T4 x2 (or 1x T4 15GB)

Approximate run time on the test set: 1hr 23 minutes

Anything else the organizers need to know: Both models are loaded in 4-bit using Unsloth to fit within the 15GB VRAM constraint. The total logical parameter count is printed at the start of the inference script for verification.

Methodology & Inspiration References

Our pipeline and human heuristics were inspired by the following resources:

• Small Language Model Reasoning: 16 Ways to make a Small Language Model think bigger (Oracle Developers)
• Knowledge Distillation: Teaching a Smaller AI Model to Think Like a Bigger One (Amol Salunke)
• Human MCQ Solving Strategies (used to inspire our inference heuristics):
  • The Art of Solving MCQs: Secrets You Must Know (Dr. Ashish Pathak)
  • MCQ Exam Tips: How to Score Higher on Multiple Choice Tests (The Study Orbit)
  • 12 Tips & Tricks to Crack MCQs (Well Guider)

Citation

This work is part of the SinhalaMMLU Shared Task. We cite the original dataset paper:

@inproceedings{pramodya-etal-2025-sinhalammlu, title = {SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala}, author = {Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro}, booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing}, year = {2025}, pages = {32943--32961}, url = {https://aclanthology.org/2025.emnlp-main.1673/} }