SinhalaMMLU-SharedTask/team-riswan

Team riswan — SinhalaMMLU Shared Task, ICTer 2026

This repository preserves Codabench submission #138, submission_think_ens_v9_final.zip, with provisional macro accuracy 64.2482 (Easy 65.2648, Medium 56.3456, Hard 71.134), as shown in the participant's 28 October 2026 screenshot. Ranking remains subject to organiser verification.

Reproduction status: the eight saved passes reproduce predictions.jsonl byte-for-byte. Fresh model inference has not been shown to reproduce the score. The supplied organiser template's no-sampling and Hugging-Face-only setup guidance differs from the historical system. See REPRODUCIBILITY.md for the full disclosure; this package does not certify rules compliance.

Team and resources

• Team: riswan. Codabench participant: riswan. The registered team name was confirmed by the participant.
• Team repository — configured repository destination; packaging changes are local until uploaded.
• Base model: google/gemma-4-E4B-it — upstream model card lists Apache 2.0.
• Ollama model: gemma4:e4b — the actual serving tag used by the original scripts; a moving tag is not an immutable model revision.
• Competition dataset.
• Task website and rules.
• No trained model or adapter is evidenced in the submitted pipeline; it uses frozen model inference. No additional model repository is currently included.

Method actually used

Two four-shot baselines (English analysis and direct-answer modes) contribute low-weight votes and identify disagreements. Five thinking passes use the original option order, cyclic rotations, or an additional terminology instruction. A separate Mac thinking run contributes another vote. A deterministic, difficulty-specific weighted merge combines the saved answers. Rotated votes are ignored when the positional-option regex matches. The entire system is not zero-shot: the four-shot examples are supplied in data/few_shot_examples.json.

Source	Saved answers	Thinking	Output-token limit	Temperature	Option order
CoT baseline	2238	false	260	0	Original; four examples
Direct baseline	2238	false	15	0	Original; four examples
Pass 1	2238	true	2048	0	Original
Pass 2	1100	true	4096	0	Rotation +1
Pass 3	720	true	4096	0.1, top_p 0.95	Original; terminology hint
Pass 4	712	true	4096	0	Rotation +2
Pass 5	712	true	4096	0	Rotation +3
Mac	1160	true	2048	0	Original

The generation requests in the archived code do not specify an Ollama seed. training/ensemble_v9.py is the exact weighted merge logic, with added validation. Ties use Counter.most_common: the first answer encountered in the fixed source order among tied totals wins. This is not an unweighted majority vote.

Parameters, hardware, and timing

• Ollama-reported parameter count: 7,996,157,674 (approximately 7.996B), comprising 7,518,069,290 main-model parameters plus 478,088,384 projector parameters. Both Kaggle servers running Ollama 0.40.12 reported these same component counts. This sum is below 8,000,000,000.
• These are model-metadata counts; no models were loaded when the diagnostic was captured. Any additional loaded components and final eligibility can be verified by the organisers during reproduction. See model_manifest.json.
• Historical hardware: Kaggle 2 × NVIDIA Tesla T4, two Ollama servers, two concurrent requests per server; separate Apple Mac with 16 GB memory reported by the participant. Exact Mac chip and historical runtime versions still need confirmation.
• The exported notebook shows Pass 4 at 106.4 minutes after 700/719 attempts, and Pass 5 at 105.6 minutes after 700/719 attempts, followed by completion messages. Each complete pass took longer than its last printed progress time. Complete end-to-end runtime was not recorded in the supplied export; no A100 timing is claimed.

Verify the archived submission without running a model

From the repository root (Python 3.10+; standard library only):

python verify_submission.py
python training/ensemble_v9.py --out predictions_v9_rebuilt.jsonl

verify_submission.py checks hashes, unique identifiers, answer ranges and byte-identical replay. The saved positional keys are archival input-derived metadata, not labels. To independently regenerate that filter and inspect target sets:

python verify_submission.py --data /path/to/dataset

This verifies the saved result; it is not a substitute for the organisers' fresh-model reproduction check.

Fresh inference reconstruction

This runner is an audited reconstruction of a multi-machine workflow, not a claim that one script originally produced v9 end-to-end. First provision the correct Ollama runtime and original model. Historical setup is preserved in the notebook under training/evidence/; it uses the Ollama installer/registry, which differs from the supplied Hugging-Face-only template. Do not silently replace the quantized model with upstream BF16 and claim equivalence. Download the complete team repository, including training/src.

pip install -r requirements.txt
mkdir -p rerun
cd rerun
python ../inference.py

The script obtains all JSON question files from DATA_REPO on Hugging Face, then uses local Ollama endpoints. Setup downloads happen before answering. For a local offline dataset use --local-data /path/to/dataset. Saved test predictions are never read by the default fresh-run path. Set OLLAMA_HOSTS to choose local servers and WORKDIR to select a fresh checkpoint directory. LIMIT=2 provides a small smoke run per difficulty; it is not a full evaluation.

The submitted predictions.jsonl is protected against overwriting. Running from rerun/ writes new answers there. A full fresh Mac pass traverses the entire priority queue: the original 1160-answer partial prefix is preserved only as historical evidence, not hard-coded as a test-count rule. Consequently the fresh pipeline is not guaranteed to produce the original votes or score.

Contents

• predictions.jsonl: exact final leaderboard predictions (2238 rows).
• inference.py, inference.ipynb: reconstructed fresh-run workflow and a notebook wrapper using the same script.
• training/ensemble_v9.py, training/passes/: exact merge logic and its eight historical inputs.
• training/kaggle_cells/, training/src/: archived stage and prompt code; archive edits are disclosed.
• training/evidence/: participant-exported notebook, original Pass 4/5 source and output logs, observed environment versions.
• data/few_shot_examples.json: the embedded labelled examples used in the prompts; source URLs were not recorded in the code.
• requirements.txt, ENVIRONMENT.md: minimal tested reconstruction dependency and historical environment evidence.
• submission_manifest.json, model_manifest.json, verify_submission.py: provenance and verification.
• REPRODUCIBILITY.md: unresolved differences, template cross-check, and verification scope.

Additional data provenance and the history of prompt/vote selection require author confirmation; absence of test gold labels in these files is not by itself proof that no test-based tuning occurred.