🧭 ScoutRoute

Can a single forward pass tell you which LLM should answer?

Logit-based competence routing: testing whether internal signals from small open-weight models predict expert competence, without RouterBench, verbal self-assessment, or sampling.

---

📌 TL;DR

A single cheap forward pass through a small generalist scout model (phi3_mini), read through a trained logistic-regression probe on its hidden states, matches — and slightly beats — a full 5-model ensemble, at ⅕ the inference cost.

	Passes required	Accuracy	Gap to oracle
Random routing	0	0.355	−0.472
Best single signal (log-prob)	5	0.650	−0.177
Domain-aware ensemble	5	0.678	−0.149
Cheap router — phi3_mini scout	1	0.684 ⭐	−0.143
Oracle (offline upper bound)	5	0.827	0.000

📄 Full write-up, per-domain breakdowns, and calibration analysis: arXiv:2610.23443.

📦 Phase 1 data is also published as a dataset: Raj-deep/scoutroute-phase1-calibration.

---

🗺️ Table of contents

• Motivation
• Models
• Datasets
• Routing signals
• Key findings
• Project structure
• Setup & usage
• Running on Kaggle
• Output files
• VRAM requirements
• Troubleshooting
• Compute budget
• Phase 1: earlier calibration study
• Citation
• License

---

🎯 Motivation

Routing between multiple LLMs usually means one of three things: an external benchmark like RouterBench, asking the model to self-report confidence in words, or sampling many completions and voting. All three are expensive, indirect, or both.

This project asks a narrower question: do the numbers already sitting inside a model's forward pass — its logits and hidden states — carry enough signal to predict which expert will get a prompt right? No extra sampling. No self-report. Just what a single pass already computes.

Five small (≤3.8B) open-weight models were pooled — three generalists, two domain specialists — and tested across four benchmark domains chosen specifically to surface real specialist-vs-generalist contrast, rather than a benchmark where every model performs identically.

---

Models

Key	HuggingFace ID	Size	Role
qwen25_1b	Qwen/Qwen2.5-1.5B-Instruct	1.5B	🔵 General
qwen25_3b	Qwen/Qwen2.5-3B-Instruct	3B	🔵 General
phi3_mini	microsoft/Phi-3-mini-4k-instruct	3.8B	🔵 General (strong for size)
qwen25_math	Qwen/Qwen2.5-Math-1.5B-Instruct	1.5B	🟣 Math specialist
qwen25_coder	Qwen/Qwen2.5-Coder-1.5B-Instruct	1.5B	🟣 Code specialist

All five are loaded in fp16 — not 4-bit. At this parameter scale, quantization overhead isn't worth it; fp16 runs faster on a T4.

⚠️ Phi-3 quirk: loaded with trust_remote_code=False. Its remote modeling code throws a rope_scaling KeyError on some transformers versions. Disabling remote code sidesteps it entirely, since Phi-3's architecture is natively supported.

---

Datasets

Four domains, loaded directly from HuggingFace with clean ground-truth answer keys — no RouterBench, no reference-answer extraction hacks.

Domain	Source	n	Why this domain
GSM8K	gsm8k (main, test)	100	Harder half only — a stratified sample from the back half of the test split. qwen25_math scores near 100% on easy GSM8K, leaving no negative examples for probe training; harder problems restore correctness variance.
HumanEval	openai_humaneval (test)	164 (all)	Code execution scoring via a subprocess sandbox. qwen25_coder expected to dominate.
MMLU	cais/mmlu (6 subjects)	100	Six deliberately diverse subjects (abstract algebra, college physics, college CS, HS world history, clinical knowledge, professional law) rather than a random sample across all 57 — avoids wasting compute on subjects with uniform difficulty.
CommonsenseQA	tau/commonsense_qa (validation)	100	No specialist advantage — the baseline domain, testing whether routing degrades gracefully when no expert has a domain edge.

Total: ~464 prompts × 5 models ≈ 2,320 (prompt, expert) pairs.

🔎 Why the validation split for CommonsenseQA?

CommonsenseQA's actual test split has withheld answer keys (reserved for a public leaderboard) — only train and validation have answerKey populated. The loader correctly uses validation.

🔎 Why the harder half of GSM8K?

Class imbalance kills probe training. If qwen25_math gets ~95% of easy GSM8K problems right, a probe trained on that data has almost no negative examples to learn from and just predicts "always correct." Taking the harder half (the later portion of the test split, which trends toward more reasoning steps) restores real variance.

---

🔬 Routing signals

Three signals, all extracted from model internals during a single forward pass — no sampling, no self-report.

Signal	Feature	How it's extracted	Training needed?
mean_log_prob	A	Average log-probability of generated answer tokens	❌
neg_entropy	B	Negated entropy of the first-token distribution	❌
probe_score	C	Logistic regression probe on the last-layer hidden state	✅ (30% of prompts)

Baselines: random routing · oracle (best expert, offline upper bound)

---

🏆 Key findings

✅	One pass matches five. A single phi3_mini forward pass (0.684) beats the full domain-aware 5-model ensemble (0.678). The cheap router isn't an approximation of the expensive method — it's competitive with it.
✅	Scout quality tracks generality, not size. Ranking: phi3_mini › qwen25_1b › qwen25_3b › qwen25_coder › qwen25_math. The two specialists are the weakest scouts — their narrow internal representations don't transfer across model families.
✅	Signals have complementary blind spots. The trained probe wins on specialist domains (GSM8K, HumanEval); raw log-prob wins on generalist domains (CommonsenseQA, MMLU). Routing by domain recovers +2.8 points over the single best signal.
✅	Calibration enables defection thresholds. Probe ECE of 0.13–0.24 vs. 0.32–0.47 for logit signals — a 3–4× improvement. That reliability made a "don't switch experts on a low-confidence margin" rule viable, recovering +4.4 points on HumanEval (0.530 → 0.574).
⚠️	The MMLU gap is structural, not algorithmic. Best score there is 0.574 vs. an oracle of 0.882 — a 31-point gap. No specialist model exists for this domain in the pool, so no routing logic can close it. Only a new MMLU-capable model would.
⚠️	The probe can still misfire. On HumanEval, it correctly detected specialist uncertainty but defected to generalists that did even worse on hard code problems — accuracy dropped to 15.6% before the defection threshold fix.

---

Project structure

ScoutRoute/
├── README.md
├── requirements.txt
├── .gitignore
│
├── src/                                 # the pipeline — run in numeric order, from the repo root
│   ├── collect_data_helpers.py          #   shared: model loading, scoring, prompt formatting
│   ├── 01_collect_data.py               #   Step 1  run inference, extract logit features
│   ├── 02a_train_probes.py              #   Step 2a train probes (baseline: full cross-domain pool)
│   ├── 02b_train_probes_specialist.py   #   Step 2b train probes (specialists trained in-domain only)
│   ├── 03_domain_aware_router.py        #   Step 3  choose the routing signal per domain
│   └── 04_cheap_router.py               #   Step 4  single-pass scout router (ScoutRoute)
│
├── notebooks/
│   └── run_experiment.ipynb             # Kaggle / Colab runner for Steps 1–4
│
├── phase1/                              # earlier calibration study (see below)
│   ├── notebooks/
│   ├── data/
│   └── results/
│
│   # ── auto-created by the scripts when you run them (git-ignored) ──
├── checkpoints/                         # JSONL, one row per completed prompt. Safe to restart.
│   └── {model}_progress.jsonl
├── hidden_states/                       # compressed numpy arrays, shape (N, hidden_dim)
│   └── {model}_hidden.npz
├── results/                             # final output tables
├── trained_models/                      # saved probe artifacts (PCA + LogisticRegression)
├── plots/                               # reliability diagrams (PNG)
└── logs/                                # per-model run logs

2a vs 2b. 02b is 02a plus one change: the qwen25_math and qwen25_coder probes train only on their own domain (GSM8K / HumanEval), which fixes a class-imbalance problem (see SPECIALIST_DOMAINS in the script). Both write to the same files in results/ and trained_models/, so run one or the other, not both.

---

🚀 Setup & usage

All commands run from the repo root.

Install

pip install -r requirements.txt

Step 1 — collect inference data

Runs each model on all four domains and saves three logit features per (prompt, model) pair. Resumable — safe to re-run after a crash; already-completed prompts are skipped.

# One model (recommended — one per Kaggle session)
python src/01_collect_data.py --model qwen25_1b

# Smoke test — small subset only
python src/01_collect_data.py --model qwen25_1b --max_prompts 50

# All models sequentially (local machine with enough GPU memory)
python src/01_collect_data.py

Recommended order (smallest/fastest first, to validate the pipeline early):

Session 1:  --model qwen25_1b      (1.5B)
Session 2:  --model qwen25_math    (1.5B)
Session 3:  --model qwen25_coder   (1.5B)
Session 4:  --model qwen25_3b      (3B)
Session 5:  --model phi3_mini      (3.8B)

Step 2 — train probes and evaluate routing

Run after all models have finished Step 1.

python src/02a_train_probes.py                 # baseline
# or
python src/02b_train_probes_specialist.py      # specialists trained in-domain only

To preview Features A and B before the probes are ready (e.g. after only one or two models have finished):

python src/02a_train_probes.py --skip_probe

Step 3 — domain-aware router

Uses the probe score on specialist domains and mean log-prob on generalist domains. Needs the probes from Step 2.

python src/03_domain_aware_router.py                       # realistic: a text classifier predicts the domain
python src/03_domain_aware_router.py --domain_mode label   # upper bound: ground-truth domain labels

Step 4 — cheap single-pass router (ScoutRoute)

One scout model runs per prompt; cross-model probes on its hidden state predict every expert's competence.

python src/04_cheap_router.py --scout phi3_mini   # the scout behind the headline result above
python src/04_cheap_router.py                     # script default scout: qwen25_1b
python src/04_cheap_router.py --sweep_scouts      # try every model as the scout

---

☁️ Running on Kaggle (recommended)

Kaggle gives 30 free GPU-hours/week with 9-hour session limits. The checkpoint system handles interruptions — restarting picks up exactly where it stopped, at the level of individual prompts. notebooks/run_experiment.ipynb wraps all of this.

First session

1. New Notebook → Settings → Accelerator: GPU T4 x1 → Internet: On
2. Get the code in: upload src/ and requirements.txt as a Kaggle Dataset, or better, clone from a private GitHub repo (see below)
3. !pip install -q -r requirements.txt
4. !python src/01_collect_data.py --model qwen25_1b
5. Before the session ends: save checkpoints/, hidden_states/, and results/ as Kaggle Output Datasets (see below)

Subsequent sessions

1. Add your saved checkpoint Datasets as input
2. Restore them into the working directory
3. Run the next model

💡 Using GitHub instead of re-uploading scripts each time

# Get the code at the start of every session
import os, subprocess
HF_USER = "Raj-deep"   
REPO = f"https://huggingface.co/{HF_USER}/ScoutRoute"
PROJECT_DIR = "/kaggle/working/project"

if os.path.exists(f"{PROJECT_DIR}/.git"):
    subprocess.run(["git", "-C", PROJECT_DIR, "pull"], check=True)
else:
    subprocess.run(["git", "clone", REPO, PROJECT_DIR], check=True)
os.chdir(PROJECT_DIR)

Code lives on GitHub, data lives on Kaggle Datasets — a clean separation. Push a fix → next session's git pull picks it up automatically.

💾 Saving checkpoints on Kaggle

Why this matters: /kaggle/working is wiped when a session ends. Without saving, every session starts from zero.

At the end of a session:

import shutil
from pathlib import Path

for subfolder, output_name in [
    ("checkpoints",   "routerbench_checkpoints"),
    ("hidden_states", "routerbench_hidden_states"),
    ("results",       "routerbench_results"),
]:
    src = f"{PROJECT_DIR}/{subfolder}"
    dst = f"/kaggle/working/{output_name}"
    if Path(src).exists():
        shutil.copytree(src, dst, dirs_exist_ok=True)
        print(f"Copied {subfolder} → {dst}")

Then in the Kaggle UI: Output tab → click each folder → Save as Dataset → name it exactly as printed (e.g. routerbench-checkpoints) → set Private → Create.

At the start of the next session:

1. + Add Data → Your Datasets → add routerbench-checkpoints, routerbench-hidden-states, routerbench-results
2. Restore them:

import shutil
from pathlib import Path

for dataset_name, subfolder in [
    ("routerbench-checkpoints",   "checkpoints"),
    ("routerbench-hidden-states", "hidden_states"),
    ("routerbench-results",       "results"),
]:
    src = f"/kaggle/input/{dataset_name}"
    dst = f"{PROJECT_DIR}/{subfolder}"
    if Path(src).exists():
        shutil.copytree(src, dst, dirs_exist_ok=True)
        print(f"Restored {dataset_name}")
    else:
        print(f"Not found: {dataset_name} (first session, or different name)")

3. Re-run src/01_collect_data.py --model <next_model> — completed prompts from prior sessions are skipped automatically; the run continues where it left off, even mid-model if a session died partway through.

📊 Checking progress without re-running inference

from pathlib import Path
for f in sorted(Path(f"{PROJECT_DIR}/checkpoints").glob("*_progress.jsonl")):
    n = sum(1 for _ in open(f))
    print(f"{f.stem.replace('_progress',''):<18} {n:>4} rows")

---

Output files explained

File	Produced by	Contents
results/{model}_results.parquet	01	One row per (prompt, model): prompt_id, eval_name, dataset, answer_key, response, is_correct, mean_log_prob, first_tok_entropy
hidden_states/{model}_hidden.npz	01	Arrays: prompt_ids (N,) and vectors (N, hidden_dim)
results/routing_results.csv	02	Accuracy for each routing signal + baselines
results/ece_results.csv	02	ECE per model per signal
results/per_dataset_routing.csv	02	Routing accuracy broken down by domain
results/probe_metrics.csv	02	AUROC, n_train, PCA dims per model probe (02b adds domain_filter)
plots/reliability_*.png	02	Reliability diagrams (predicted signal vs. actual accuracy)
trained_models/{model}_probe.pkl	02	Saved PCA + LogisticRegression probe (joblib)
results/domain_aware_routing.csv	03	Full domain-aware comparison table
results/domain_classifier.csv	03	Domain classifier accuracy (predict mode)
trained_models/domain_clf.pkl	03	Saved TF-IDF domain classifier (predict mode)
results/cheap_router_results.csv	04	Full cheap-router comparison table
results/cheap_probe_metrics.csv	04	AUROC per (scout → target) probe
trained_models/cheap_probes/	04	Saved cross-model probes, one per target expert

---

VRAM requirements (fp16)

Model	Params	VRAM
qwen25_1b, qwen25_math, qwen25_coder	1.5B	~3 GB
qwen25_3b	3B	~6 GB
phi3_mini	3.8B	~8 GB

A T4 (16 GB) handles all five models comfortably in fp16.

---

🛠️ Troubleshooting

"No *_results.parquet found"

Step 1 hasn't completed for any model. Run src/01_collect_data.py first.

Phi-3 fails to load / rope_scaling KeyError

Already handled — TRUST_REMOTE_CODE["phi3_mini"] = False in src/01_collect_data.py. If you see this error anyway, check your transformers version is recent (pip install -U transformers).

Phi-3 appears stuck / inference is extremely slow

Check it actually loaded on GPU, not CPU. device_map={"": 0} is used explicitly (not "auto") because "auto" has been observed silently placing Phi-3 on CPU on some setups.

HumanEval scores look too low

Check max_new_tokens — code solutions need more room than other domains. This is already handled (run_inference bumps to 256 tokens for dataset="humaneval" automatically), but if you've modified the script, confirm that bump is still in place.

"Too few training samples — probe skipped"

That model's training split (30% of ~464 prompts ≈ 140, then split again 80/20 inside train_probe) has fewer than 30 examples — usually because Step 1 didn't fully complete for that model. Check checkpoints/{model}_progress.jsonl row count. With 02b, the specialist probes train on in-domain prompts only, so their pools are smaller still.

Kaggle session killed mid-run

Restore checkpoints (see above), re-run the same --model command. The script reads the JSONL checkpoint, skips completed prompts, and continues.

---

Estimated compute budget

Model	Domain prompts	Approx. GPU time (T4, fp16)
qwen25_1b	464	~1.5 hrs
qwen25_math	464	~1.5 hrs
qwen25_coder	464	~1.5 hrs
qwen25_3b	464	~2.5 hrs
phi3_mini	464	~3 hrs
Total		~10 GPU-hours

Comfortably fits in a single week on Kaggle's free 30 GPU-hour/week tier — likely achievable in 2–3 sessions.

---

🧪 Phase 1: earlier calibration study

phase1/ holds the exploratory study that came before the pipeline above: it compared verbalised confidence against answer-token confidence across the same five experts, and prototyped routers and a cascade on top. It is kept for reference and reproducibility; the pivot away from verbal self-assessment is what led to the logit-based approach in this repo.

---

📖 Citation

If you use this work, please cite the paper:

@misc{pandhare2026scoutroute,
  title         = {ScoutRoute: Efficient Mixture-of-Experts Routing Using Internal Language Model Signals},
  author        = {Pandhare, Rajdeep and Bharucha, Delzaan},
  year          = {2026},
  eprint        = {2610.23443},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2610.23443}
}

---

License

Released under the MIT License.

_{Built with 5 small open-weight models, 4 benchmarks, and zero extra sampling.}