SinhalaMMLU-SharedTask/team-hakuru-labs-helalm

HelaLM-4B SinhalaMMLU Adapter (v2)

This is a LoRA adapter fine-tuned for the ICTer 2026 SinhalaMMLU Shared Task.

Model Details

• Base Model: Jainamshahhh/helalm-4b (which is a Sinhala-tuned Gemma-3-4B model).
• Architecture: Gemma 3 (4B)
• Adapter Size: ~24M parameters
• Total Logical Parameters: ~4.02B (Well under the 8B competition limit)
• Language: Sinhala (සිංහල)

Training Data & Methodology

The model was fine-tuned on a custom dataset of ~13,500 Sinhala multiple-choice questions.

• Synthetic Data: Generated using closed APIs (DeepSeek, Claude, and ChatGPT) from official Sri Lankan government school textbooks (Grades 6–13) and teacher guides. (Disclosed in accordance with competition rules).
• Source Material: Downloaded from GovDoc.lk (https://govdoc.lk/). A full breakdown of the specific subjects, textbooks, and teacher guides used is provided in the main team repository.
• Public Data: The official SinhalaMMLU Easy-tier development set.
• Training Format: This adapter was fine-tuned using the Alpaca instruction format, not Gemma's native chat template. It learns to directly output the correct answer token (පිළිතුර: X).

Citation

This work is part of the SinhalaMMLU Shared Task. If you use this model, please cite the original dataset paper:

@inproceedings{pramodya-etal-2025-sinhalammlu, title = {SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala}, author = {Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and Kamigaito, Hidetaka and Watanabe, Taro}, booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing}, year = {2025}, pages = {32943--32961}, url = {https://aclanthology.org/2025.emnlp-main.1673/} }

Usage

To use this adapter, you must load the base model first.

Important: This adapter was fine-tuned using the Alpaca format. You must use this exact format at inference.

Prompt template

### Instruction:
පහත බහුවරණ ප්‍රශ්නයට නිවැරදි පිළිතුර තෝරන්න. පළමුව සෑම විකල්පයක් ගැන සිතන්න, පසුව පිළිතුර ලෙස A, B, C, හෝ D පමණක් ලියන්න.

### Input:
[ප්‍රශ්නය]
A) [විකල්පය A]
B) [විකල්පය B]
C) [විකල්පය C]
D) [විකල්පය D]

### Response:
පිළිතුර:

Code

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = "Jainamshahhh/helalm-4b"
adapter_model = "SinhalaMMLU-SharedTask/team-hakuru-labs-helalm" 

# Load base model in 4-bit to save memory
tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
    base_model, 
    load_in_4bit=True,
    torch_dtype=torch.float16
)

# Attach the LoRA adapter
model = PeftModel.from_pretrained(model, adapter_model)
model = model.to(torch.float16) # Ensure dtype match for Gemma architecture