SinhalaMMLU-SharedTask/team-hakuru-labs-redqueen

RedQueen v7 - SinhalaMMLU Shared Task Adapter

This is a LoRA adapter fine-tuned for the ICTer 2026 SinhalaMMLU Shared Task.

Model Details

• Base Model: iCIIT/redqueenprotocol-sin-llama3.2-3B-model
• Architecture: Llama 3.2 (3B)
• Adapter Size: ~24M parameters
• Total Logical Parameters: ~3.22B (Well under the 8B competition limit)
• Language: Sinhala (සිංහල)

Training Data

The model was fine-tuned on a scaled-up custom dataset of ~24,500 Sinhala multiple-choice questions.

• Synthetic Data: Generated using closed APIs (DeepSeek, Claude, and ChatGPT) from official Sri Lankan government school textbooks (Grades 6–13) and teacher guides. (Disclosed in accordance with competition rules).
• Source Material: Downloaded from GovDoc.lk (https://govdoc.lk/). A full breakdown of the specific subjects, textbooks, and teacher guides used is provided in the main team repository.
• Public Data: The official SinhalaMMLU Easy-tier development set.
• Training Format: While the system prompt instructs the model to "think step-by-step", the actual training targets contain no explicit Chain-of-Thought (CoT) text. The model is trained to directly output the final answer token (පිළිතුර: X), forcing it to reason internally.

Citation

This work is part of the SinhalaMMLU Shared Task. If you use this model, please cite the original dataset paper:

@inproceedings{pramodya-etal-2025-sinhalammlu, title = {SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala}, author = {Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro}, booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing}, year = {2025}, pages = {32943--32961}, url = {https://aclanthology.org/2025.emnlp-main.1673/} }

Usage

To use this adapter, you must load the base model first:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "iCIIT/redqueenprotocol-sin-llama3.2-3B-model"
adapter_model = "SinhalaMMLU-SharedTask/team-hakuru-labs-redqueen" 

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model, load_in_4bit=True)
model = PeftModel.from_pretrained(model, adapter_model)