SinhalaMMLU-SharedTask/team-ctrl-llama-lora

CTRL — Llama 3.2 3B LoRA adapter

An answer-choice adapter for Sinhala multiple-choice questions. This repository contains adapter weights and the exported tokenizer; the base weights are loaded separately.

Members: Jayendra Matarage · Prashan Samarathunge · Isuru Udayanga · Mithun Nimesh

Team code, inference and predictions · Adapter repository

Architecture

A frozen Llama 3.2 3B Instruct base with one PEFT LoRA adapter, loaded through Unsloth in NF4 4-bit with double quantization and bfloat16 compute.

Component	Configuration
Llama decoder	28 layers, hidden size 3,072, MLP 8,192; 24 attention / 8 KV heads
LoRA	Rank 16, alpha 32, dropout 0; attention q/k/v/o and MLP gate/up/down projections
Parameters	3,212,749,824 base + 24,313,856 adapter = 3,237,063,680 total
Base revision	bb1d317a108579fb40e646af8924a5e7ec5604b1

Counts use the saved architecture and adapter tensor shapes; quantization does not reduce parameter count.

Inference: question + choices → chat prompt → base + LoRA → logits for 1–4 → argmax. The exported tokenizer/chat template is retained; scoring uses no generation, sampling, retrieval or hosted API.

Training

Three epochs on 3,968 labeled rows using four-choice cross-entropy, AdamW 8-bit, LR 1e-4, cosine scheduling and 3% warmup. Batch 1 × accumulation 8, context 8,192, bfloat16 and gradient checkpointing; seed 42. Only LoRA weights were updated.

Selection used training-fit logit_accuracy (47.91%) on those same rows, not held-out validation; export: checkpoint-1488. CPT, synthetic training and the optional continuation were not used. See the team training description for settings, fit diagnostics and optional textbook materials.

Data provenance and limitation

Training used 2,264 labels with closed-LLM provenance and 1,704 supplied labels. Inputs include competition test questions labeled/validated using closed LLMs, conflicting with the prohibition on test-question training. Excluding their files does not undo training; organizer judgment is required. Original metadata preserves this lineage.

Task questions/fit dumps are excluded; original easy-eval remains local. Sources: rules, task dataset, public SinhalaMMLU, dataset paper. Optional textbook materials retain source/generation provenance, without a new redistribution license.

Loading and artifacts

Run inference.ipynb using the team instructions. It downloads the pinned base, uses the exported configuration/tokenizer, and attaches this adapter once with PeftModel.from_pretrained(..., is_trainable=False). Loaded adapter tensors are checked; weights are not merged or requantized.

Weights SHA-256: 1ebee225446871aeb74215ee38554d39b2259f5182acd974234bc70e6f3ff812.

Weights/tokenizer/configuration are unchanged. adapter_integrity.json records hashes; run_config.json, dataset_identity.json, training_provenance.json and gold_fit_summary.json preserve the run record. Versions: installed_versions.json, pip-freeze.txt. Per-question fit predictions and pickled training arguments are omitted. Llama 3.2 license/use conditions apply.

Environment and verification

Recorded training: H100 80GB HBM3, Python 3.12.3, Torch 2.8.0+cu128, CUDA 12.8; 4,146.525 seconds (69.11 minutes). Test-inference runtime is unavailable.

Checked: notebook/data contracts, all 2,238 prompt lengths, tokenizer answer boundaries, Linux dependency resolution and remote hashes. Full installation, CUDA loading and reproduction comparison remain unverified. Both repositories require authorized Hugging Face access.