HER Hack-Astron #6 | Spark-X2.5-4B MATH-500 evaluation (60 stratified, greedy pass@1, 61.7%)
This is my entry for the HER Hack-Astron #6 challenge. I ran the official XHToken/Spark-X2.5-4B checkpoint (revision 5e10fcc0286756aebf7c41dc52c1e42d95c70281) on a stratified 60-problem slice of MATH-500 (test split), with greedy decoding (pass@1) and thinking mode enabled through the official chat template.
Results at a glance
Metric Value
Accuracy (pass@1, greedy) 37 / 60 = 61.67%
Unparseable outputs (no final answer extracted) 14 / 60
Generation mode greedy, temperature 0, max_new_tokens 4096
Precision fp16 (no quantization)
Hardware Google Colab free tier, Tesla T4 (16 GB)
Total wall time ~132 min
Setup and reproducibility
Model: XHToken/Spark-X2.5-4B, revision 5e10fcc0286756aebf7c41dc52c1e42d95c70281 (official HF checkpoint), loaded with trust_remote_code=True.
Precision: fp16 (torch_dtype=torch.float16). One thing worth flagging: bitsandbytes 4-bit quantization is not compatible with this model's custom modeling code — the code casts hidden states to the weight dtype, so under 4-bit everything becomes torch.uint8 and it dies with baddbmm_cuda not implemented for 'Byte'. fp16 sidesteps that, needs no quantization, and fits in 16 GB VRAM (weights ~8.2 GB).
Transformers: 4.57.1 (the version declared in the model config; transformers 5.x breaks the custom _tied_weights_keys list format with 'list' object has no attribute 'keys').
Dataset: HuggingFaceH4/MATH-500 test split, 60 problems, stratified by difficulty (level 1–5) with seed 42. Distribution: L1: 5, L2: 11, L3: 13, L4: 15, L5: 16.
Prompt: official chat template with enable_thinking=True; system message "You are a helpful math assistant."; user message "Solve the following math problem. Reason step by step, then give your final answer inside \boxed{} at the end." + problem.
Decoding: max_new_tokens=4096, do_sample=False, temperature 0 (greedy, single sample per problem).
Output: per-problem JSONL records (problem, full raw output, extracted answer, gold answer, grading method, correctness, generation time) plus a summary.json. Script and judge pipeline are included below.
How I graded
Extract the final answer from the raw output (priority: first \boxed{...} with nested-brace counting, then explicit "the answer is" patterns, then trailing content after the last =).
Grade with a three-level equivalence check:
numeric: parse decimals / fractions / percentages, compare with math.isclose (rel/abs tol 1e-6)
symbolic: LaTeX → sympy normalization (\frac{a}{b}→(a)/(b), \sqrt{a}→sqrt(a), \pi→pi, …), compare simplify(pred − gold) == 0
string: whitespace/punctuation/dollar-normalized exact match
If no answer can be extracted, the item is marked unparsed (counted as incorrect, reported separately).
The judge was validated locally before the run: 28 unit tests + 40 real-pipeline tests (LaTeX fractions, radicals, π, polynomials, numeric tolerance) — all passing.
Results by difficulty
Level Accuracy
1 2/5 = 40%
2 7/11 = 64%
3 11/13 = 85%
4 9/15 = 60%
5 8/16 = 50%
Results by subject
Subject Accuracy
Algebra 9/14 = 64%
Counting & Probability 1/3 = 33%
Geometry 0/4 = 0%
Intermediate Algebra 8/13 = 62%
Number Theory 4/5 = 80%
Prealgebra 11/16 = 69%
Precalculus 4/5 = 80%
Failure analysis
The 23 wrong items fall into three buckets.
A. Truncated thinking chains (~11 items). On the hardest problems (mostly L4–L5) the reasoning exceeds max_new_tokens=4096 and generation is cut off before a final answer appears. These items time out at ~300 s and yield nothing extractable. I read this as a faithful upper-limit signal of how long the model's reasoning actually gets, not a parse failure.
B. Judge misses on numerically/semantically correct answers (2 items). Two examples:
[19] model produced 4210_5; gold is 4210_{5} — same base-5 number, only the brace formatting differs.
[59] model produced 15; gold is 15 \mbox{ cm}^2 — the value is correct, only the unit is missing.
C. Genuine reasoning errors (~9 items). Representative examples:
A region/area problem where the model returned an expression x-1 instead of the value \frac{1}{2}.
A simplification problem where the model answered t+1 instead of 2.
Notes
Dataset split, the exact 60-problem subset (seed 42 stratified), judge logic, unparseable handling, and the full per-item JSONL are all documented above, so the numbers can be verified directly.
The 14 unparseable items stay in the denominator (60) and are scored as incorrect.
The judge and the complete evaluation script are attached to this post for reproduction.
A few things I noticed along the way:
Spark-X2.5's thinking mode is very effective on routine algebra / prealgebra but often runs out of budget on hard geometry and multi-step combinatorics.
Since the model was run in fp16 (no quantization), these numbers reflect the original checkpoint without any quantization loss.
Reproduction
Full self-contained script (spark_math_eval.py), judge modules, and tests are attached to this post. Steps:
!pip install -q "transformers==4.57.1" accelerate datasets sympy
Upload spark_math_eval.py to /content/
!python spark_math_eval.py --dry-run (1-problem smoke test)
!python spark_math_eval.py --out-dir /content/drive/MyDrive/spark_eval_out (60 problems)
All outputs (per-item JSONL + summary) are saved incrementally, so a dropped Colab session resumes without losing work.
— liuxxxs