ITheEqualizer/gsm8k-reasoning-depth-router

GSM8K Reasoning Depth Router v1.0.0

This compact advisory classifier predicts whether an English grade-school math word problem is likely to need standard reasoning (0–3 annotated calculations) or extended reasoning (4 or more). Maintainer: Ali Zakaee (ITheEqualizer).

Intended decision and audience

The named consumer is an evaluation or tutoring pipeline that repeatedly chooses between a standard and an extended reasoning budget before solving a problem. The model does not solve math, grade answers, or authorize spending. It only returns a score and suggested budget.

Positive extended example: calculate a test score from three question types, three success percentages, and different point values. Negative standard example: 12 pencils are shared equally among three students. Empty and over-800 character inputs are rejected.

Tested quickstart

python -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m gsm8k_reasoning_depth_router.predict \
  "Twelve pencils are shared equally among three students. How many does each get?"
.venv/bin/python -m gsm8k_reasoning_depth_router.reference_consumer \
  "Amy gets 80% of 10, 90% of 20, and 60% of five 5-point questions. What is her score?"

The reference consumer maps standard to use_standard_reasoning and extended to request_extended_reasoning; it performs no external action.

Data, labels, rights, and leakage

Training uses the human-written GSM8K main configuration at Hub revision 740312add88f781978c0658806c59bc2815b9866, derived from the official MIT source at commit 3101c7d5072418e28b9008a6636bde82a006892c. Contractors wrote problems and solutions, and different workers independently re-solved them; the authors estimate about 1.7% may still contain errors or ambiguity.

The target is derived from the gold solution: extended means at least four <<calculation>> annotations. It is not an LLM-generated label. The 7,473-row development set contains 4,814 standard and 2,659 extended examples. The official 1,319-row test lockbox contains 804 standard and 515 extended examples. Digits are normalized before group hashing; there are zero normalized-template collisions between development and lockbox. An 80/20 grouped development split is used for fitting and validation-only blend/threshold selection. The official test is untouched until the rule is fixed.

Representation and training

The lexical component hashes word 1–2 grams into 4,096 bins and character 3–5 grams into 8,192 bins. An averaged SGD logistic classifier uses balanced class weights, alpha=2e-5, seed 20260813, and at most 800 iterations. A second logistic classifier learns five explicit shape features: character count, token count, number count, each/every/per cues, and arithmetic-symbol count.

Validation selects a 50/50 score blend and threshold 0.4506788131 by maximizing macro-F1 subject to at least 0.65 recall for both labels. This representation was chosen after the lexical-only fit missed extended problems. The learned-vocabulary word-TFIDF baseline reached only 0.6526 lockbox macro-F1; the simple shape baseline reached 0.6807; majority prediction reached 0.3787.

Measured results

Validation macro-F1 is 0.7252. On the untouched lockbox, macro-F1 is 0.7212 (bootstrap 95% interval 0.6966–0.7453), accuracy is 0.7271, and ROC-AUC is 0.8041. Confusion counts are TN=575, FP=229, FN=131, TP=384. Extended recall is 0.7456 and precision is 0.6264; standard recall is 0.7152 and precision is 0.8144. The 325,519-byte artifact is 79% smaller than the 2 MB gate and has an empty skops untrusted-type set. Complete metrics are in metrics.json.

False extended routes spend extra compute but still solve normally. False standard routes are more serious because they can under-allocate reasoning; 131 extended lockbox problems were missed. The confidence interval overlaps the 0.72 release threshold, so treat the point estimate as modest evidence, not a guarantee of broad generalization.

Prediction trace

Raw text is whitespace-normalized and bounded. The same question feeds the fixed-width lexical hasher and five shape features. Each learned classifier returns an extended score; the 50/50 blend is compared with 0.4506788131. Scores at or above the threshold return extended and a suggested budget of four; lower scores return standard and a suggested budget of one.

Reproduction

dataset_manifest.json binds both Parquet files to SHA-256. Run gsm8k_reasoning_depth_router.release_train with those files and reproduction_reference.npz; every lockbox score, prediction, and label must match exactly. The release passed exact clean refitting, serialization round-trip, static analysis, tests, checksums, and a fresh-environment inference quickstart.

Limitations, integration, and rollback

• This is a routing proxy for annotated calculation count, not actual solver difficulty, correctness, latency, or cost.
• Results cover English GSM8K-style elementary arithmetic, not algebra proofs, multilingual questions, OCR text, adversarial inputs, or real tutoring traffic.
• Fixed hashing creates collisions and limits feature-level explanations.
• The annotation-count boundary is useful operationally but not a universal definition of reasoning complexity.
• Use the result only as an advisory budget hint. Measure downstream task success and latency on local data; retain a fallback that always requests extended reasoning when uncertainty or impact is high.

Rollback is to disable the router and send every problem through the existing extended-reasoning path. Verify model.skops against config.json, load with trusted=[], and reject schema, checksum, label-map, empty-input, and length mismatches.