Pricile/cassava-rca-detective

1st place — Cassava AI Root Cause Detective

5G root-cause diagnosis · ≤ 4B open model · Solution writeup · 1st prize

Zindi "Cassava AI Root Cause Detective Hackathon" · Deep Learning Indaba 2026 · public 0.99 (258/259) · 1st prize ($1000).

---

🙏 Acknowledgements

Before anything else, a heartfelt thank-you to my mentor Kevin Degila for his mentorship, his advice throughout this challenge, and for providing the compute resources that made these training runs possible. This result would not have been possible without his guidance and support. 🙏

---

🟢 Background — the honest story

A short bit of story, because it drove every technical decision.

I started this challenge with a strong-looking neuro-symbolic idea and reached ~0.97 public fast — by injecting the measurements my rule engine computed into the prompt and training on pseudo-labels of the test. Then, on 23 July, Zindi clarified the rules: the model must generate the answer, no rule-based computation, no test↔train/validation lookup. My 0.97 method was now non-compliant. I had to throw it away.

What actually won this competition is what came next: rebuilding a fully compliant pipeline and then measuring, not guessing, my way up from 0.84 to 0.996. Every jump below is a measured number, and the most important tool I built was not a model it was a way to predict my public score before submitting. Below is the full method, with numbers, and the dead ends too.

---

🥇 Winning solution

TLDR: LoRA fine-tune of an open-source ≤ 4B model (Qwen3-4B-Instruct-2507) on reasoning traces written by my own pure-Python rule engine (no LLM in the data pipeline). At inference the model reads the raw question and generates its own reasoning + \boxed{} answer. Questions are then routed by family, with a small unlabelled self-consistency cascade and two more ≤ 4B models arbitrating the hardest general-knowledge MCQs.

• Base model: Qwen/Qwen3-4B-Instruct-2507 (open weights, Apache-2.0, run locally). Fits the ≤ 4B rule (declared honestly: 4,022,468,096 total params, 3.63 B excluding tied embeddings).
• Training data — the compliant core: the test splits into 3 families — 681 telelogs "C" questions (8 causes), 100 signalling "A-I" questions (9 causes), 82 general-knowledge MCQs. I built the SFT reasoning traces with two hand-written rule engines (rule_solver.py, ai_solver.py, verified 2400/2400 on train and 864/864 on validation), from the provided data only + synthetic A-I clones I generate myself. No language model of any size touches the data pipeline — a real advantage, since the reference paper distils traces from 32B models, which the rules later disallowed.
• The key training idea — "decisive" traces: my first traces marked several candidate causes as SUPPORTED, and the model learned to hesitate. I forced exactly one cause SUPPORTED per trace (the true root cause; near-misses explicitly ruled out). C-family agreement jumped 0.82 → 0.956.
• Fine-tuning: LoRA r64, 2 epochs, lr 1.5e-4, effective batch 16, bf16, gradient checkpointing, seed 42, on 1× NVIDIA L40S (48 GB).
• Inference: greedy, deterministic, raw question in → \boxed{} out. No measurement/threshold in the prompt, no rule engine at inference.
• Assembly (build_submission_v6.py): route by family; unlabelled self-consistency repairs (hesitation fallback + arithmetic detectors); MCQ cascade. It reads only test.csv + SampleSubmission.csv + model generations — no label file.

---

📈 What actually moved the score [every number measured]

Step	Lever	Public
Compliant baseline	ambiguous traces, single model	0.8378
Decisive traces + family routing	one SUPPORTED per trace; C←decisive model, A-I←other model	0.9537
Hesitation fallback	if the model marks ≠ 1 SUPPORTED, take another model's generation (+12 questions)	0.9691
Arithmetic self-consistency	reject generations whose own numbers (mod-30, thresholds, downtilt) contradict the verdict; repair from a model retrained to state them correctly	0.9846
MCQ cascade	self-consistency vote → prompt-ensemble → un-fine-tuned base model arbiter → Qwen3-4B-Thinking-2507 reasoning arbiter on the hardest contest problems	0.9884
Final MCQ reasoning + selection	reasoning model resolves the last undecided MCQs; two finals bracket the one publicly-unmeasured question	0.9961

• Decisive traces — the biggest single lever. The rule engine solves the C-family at 100 %, so the answer is in the question; a model stuck at 0.82 was executing the reasoning sloppily, not missing information. Teaching it to commit to exactly one cause moved C from 0.82 → 0.956. Error analysis proved it: when the model marks 1 SUPPORTED it is 95.4 % right; with ≥ 2, ~60 %.
• Routing, not a single model. Folding decisive-C and A-I into one model broke A-I badly (dilution). Routing C→decisive model and A-I→the other model kept both (C 0.956 + A-I 0.96) — compliant, since routing is by question format, exactly like general-MCQ→base.
• Detectors that read the model, not the data. The model sometimes asserts a false congruence ("PCI 745 vs 980: both mod 30 = 20", while 745 % 30 = 25). A detector that checks the arithmetic the model itself wrote fired 18 times, 18 wrong, 0 correct broken. Root cause was in my training corpus (no negative example of "moduli differ") — fixed by patching the trace generator, not the model.
• The MCQ contest problems needed a reasoning model. The base model returned 0 readable answer in 32 draws on some problems — not ignorance, truncation (it never reached \boxed{} in 1200 tokens). At ≥ 4096 tokens, Qwen3-4B-Thinking-2507 finishes them.

---

🔬 The thing I'm most proud of: measuring without test labels

With no test transcripts, I built a test-proxy: run inference on test.csv, score the agreement with my rule engine's predictions (oracle_test.csv + ai_predictions.csv). It predicted the public leaderboard to ±0.001 every time (0.837 test-proxy → 0.837838 public; 0.956 test-proxy → 0.953667 public). That let me iterate for days without spending a single submission, and select models on a faithful signal instead of leaderboard feedback.

A hierarchy of measurement instruments, by trust: (1) the rule-engine test-proxy on the measurable families; (2) the model's own self-consistency (SUPPORTED count, arithmetic checks); (3) the public leaderboard (costs a slot, and only 259/863 questions). Most mistakes came from trusting a lower rung than I should have.

---

🧱 Dead ends (all measured, none guessed)

• Single unified model (decisive C + rebalanced A-I). Retrained with A-I at 12 % of the mix; A-I still collapsed to 0.10 and C regressed to 0.87. Routing two models beats one here. Kept the routing.
• k = 8 self-consistency on the decisive model. Test-proxy +0.15 pt on C, but public went down (247→245/259). At the ceiling, this is noise. Kept greedy.
• The 3 last measurable errors (2 C, 1 A-I where the model disagrees with the rule engine). My fine-tuned model, the base model and the reasoning model each give a different answer — genuinely ambiguous. No compliant way to fix them, so I left them.
• Evidence injection (the old 0.97). Banned by the 23 July clarification. Abandoned entirely and documented as non-compliant.

---

🔁 Reproducibility & compliance

• Byte-exact reproduction, no GPU. The model generations are frozen in the package; verify_reproduction.py extracts it, adds only test.csv + SampleSubmission.csv, replays the assembly, and confirms the two selected files are regenerated byte-for-byte in under a minute.
• Compliance, rule by rule (CERTIFICAT_CONFORMITE.md): ≤ 4B open model; every answer generated by the model from the raw question; no rule engine at inference; no test lookup; provided data only; no LLM in the data pipeline. I also self-declare one defect I found and fixed (2 of 863 answers had been written by a format guardrail rather than the model — regenerated by sampling).
• Fixed seeds, pinned requirements.txt, documentation following the Zindi guidelines (SOLUTION_DOCUMENTATION.md).

---

🔎 If I had more time

• Train longer (the schedule was short) and try syllable/BPE tokenisation for the reasoning traces.
• A cleaner single model for A-I, so routing isn't needed.
• A larger clean dev to tune MCQ decoding at higher statistical power.

---

🗝️ If one idea survives this writeup

Measure, don't guess — and audit your own training data. My biggest jump (decisive traces) came from looking at what my traces were teaching the model, not from a bigger model or more data. And the test-proxy that predicted the public score to ±0.001 is what made every other decision safe.

---

Author: Pricile GANGBE (Zindi: Pricilegangbe). Everything Apache-2.0 / open. Code, models and the reproducible review package are in this repository.