SinhalaMMLU-SharedTask/team-AlphaPack

Team AlphaPack — SinhalaMMLU Gemma 4 E4B V2

Status: audited submission CANDIDATE; full clean-room prediction reproduction NOT yet demonstrated. Team identifier: AlphaPack. Before submission: add your real team-member details and confirm there are no team adapters/extra training assets omitted.

Submission

• Base model: google/gemma-4-E4B-it
• Revision: ee0ef6023621cff504d758262d4e04895a5af4a2
• Unique inference-time parameter count verified in the source recovery notebook: 7,941,100,832 (7.9411B; below 8,000,000,000), including vision/audio modules. No additional trainable inference adapter is included in this package.
• Technique: native Gemma 4 thinking (enable_thinking=True) + exact V2 explanation-first / last-line ANSWER: X prompt, with automatic invalid-answer retries. The text is SHA-256 pinned in inference.py.
• Decoding: greedy (do_sample=False); first pass batch size 4, 2,048 new tokens; strict targeted retries batch size 1, 4,096 new tokens. The first-pass settings are reconstructed from research notebook material, not independently proven identical to the historical all-2,206 run.
• Hardware used for historical research: Kaggle NVIDIA T4 ×2; FP16; SDPA. Original targeted retry set a four-hour run limit, not a measured full-set runtime. Actual complete fresh-run runtime: not yet measured.
• Seed: 2026 (reconstructed setting; confirm against initial production records).
• Dataset used for evaluation: SinhalaMMLU Shared Task public questions, downloaded by the script from DATA_REPO.
• Additional training / retrieval data: none in this system package. Confirm any other teammates' work before publication.
• Team member details: Team AlphaPack — Umesh Yapa, Esandu Epa, Lasal jayasinghe, Amasha Ranasinghe.

Files

• inference.py — best-effort self-contained fresh inference candidate; loads every .json in DATA_REPO, downloads pinned model from HF, uses native thinking and V2 prompt, retries invalid answers, and writes predictions.jsonl in the required numeric-label format.
• predictions.jsonl — unchanged original user-provided 2,238-row final submission candidate; SHA-256 dcf8cf70066f06b56c319a5eb6604bbeb823cc70096ab8a5c40e8c6d05e06b00.
• requirements.txt — reconstructed package versions from the research environment.
• recovery/evidence/ — independently verified historical first-pass 2,206 + 32 failures package, merge audit, hashes and merge source code.
• recovery/MIDDLE_E4B_RETRY32_COLDSTART.ipynb and .py — verbatim original 32-question historical recovery code, requiring the original combined-2,206 package and 41 unchanged JSON question files as inputs. Its source contains the final strict parser, per-question logs, model hash, dataset checks and successful-only final export logic. This notebook has no embedded execution outputs.
• training/README.md and data/README.md — no fine-tuning, adapters or extra training data claimed; review if other team components were used.
• REPRODUCIBILITY_STATUS.md — limitations and outstanding evidence.
• check_submission.py, verify_final_predictions.py — file integrity / output validation helpers.

Fresh environment reproduction

1. On Kaggle, choose GPU T4 ×2, enable Internet, and accept the Gemma license through your Hugging Face account if required. The model is gated. Authenticate externally through HF_TOKEN or pre-existing Hugging Face login; never place tokens in code.
2. Install pip install -r requirements.txt.
3. The organizers change only DATA_REPO in inference.py; then run python inference.py from a clean working directory. It downloads the model from Hugging Face and produces predictions.jsonl.
4. Keep an immutable copy of this repository's original predictions outside the run directory. Running the script overwrites the predictions.jsonl in its working directory.
5. Compare the reproduced predictions to the original 2,238-row file with python check_submission.py ORIGINAL.jsonl GENERATED.jsonl. Byte-identical SHA is the strongest evidence; report any differences, never conceal them.

Critical reproducibility limitations

The original E4B_COMBINED_2206_WITH_FAILED32.zip is now included under recovery/evidence/. It provides the original 2,206 valid predictions, 32 failed attempts, 32 raw failure outputs, merged first-pass audit, dataset SHA-256 manifest and original merge script. An independent offline audit verified all 2,206 entries match predictions.jsonl exactly and the remaining 32 indices are disjoint and have matching question metadata. The first-pass merge audit confirms the same V2 prompt hash as RETRY32. Three original shard ZIPs are still not provided, although their names and SHA-256 hashes are recorded. The complete original first-pass generator is still not available as one final standalone script; the integrated inference.py is reconstructed, and a full clean-room GPU run has not been executed here. Clean-room inference has not been run or checked against the exact submitted file here. This README is intentionally not a claim of exact output reproducibility.

Required repository links

• System repository: https://huggingface.co/SinhalaMMLU-SharedTask/team-AlphaPack
• Model weights: https://huggingface.co/google/gemma-4-E4B-it — pinned revision above; no fine-tuned adapter claimed.