team-ape-llama: SinhalaMMLU shared task (ICTer 2026)

• Code repo (this): https://huggingface.co/SinhalaMMLU-SharedTask/team-ape-llama

• Adapter repo: https://huggingface.co/SinhalaMMLU-SharedTask/team-ape-llama-gemma4-e4b-lora

• Run: pip install -r requirements.txt && python inference.py --input <questions folder or .jsonl> --output predictions.jsonl (--perm 1 = single pass)

• Base model: google/gemma-4-e4b-it (text-only load; per-layer embeddings counted): 7.480 B parameters, loaded in bf16.

• Adapter (single LoRA): 17,440,768 parameters (r=8, all attention + MLP projections). Repo: https://huggingface.co/SinhalaMMLU-SharedTask/team-ape-llama-gemma4-e4b-lora.

• Total inference parameters: about 7.498 B (<= 8 B). One base model + one adapter. No ensemble, no router, no retrieval, no internet, no closed API at inference.

• Prompt: Sinhala instruction (si_basic), zero-shot, logit scoring over the option labels 1-4.

• perm4 (what we submitted): deterministic option rotation: the same question is scored 4 times with the options in a fixed rotated order and the scores are summed. Same model, same adapter, no extra parameters; no sampling (do_sample=False), fixed seed.

• Codabench: Easy 52.0249 / Medium 49.2349 / Hard 57.7320 / Macro 52.9973 (predictions.jsonl in this repo is the exact file that was uploaded).

• Hardware / runtime: 1x NVIDIA A100 (Google Colab); test set (2,238 questions) scored in about 285 s with perm4, plus model download.

• Extra data used: see training/README_training.md (composition of one data file inferred from file sizes, to be confirmed).

• Training: training/.

• Full code and MANIFEST: private GitHub repo, tag final-oct28 (branch final/oct28), configs configs/sub_run2_bf16_perm4.yaml (and ..._perm1.yaml).

• Citation: Pramodya et al., SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala, EMNLP 2025, pp. 32943-32961.