Title: Puzzle Everything: Benchmarking Knowledge Reasoning through Generalized Puzzle Solving
Primary area: datasets and benchmarks
Keywords: Large Language Models, Data Leakage, Evaluation, Puzzle

Abstract: The validity of existing benchmarks is challenged by data leakage, which risks reducing current evaluation from genuine problem-solving to mere recitation. This concern highlights a fundamental need for robust methods to assess reasoning based on a model's internalized knowledge. To address this issue, we introduce the Puzzle Everything Benchmark (PEB), which fundamentally expands the puzzle-solving application for reasoning evaluation. By treating any learned concept, term, or entity as a solvable puzzle, our approach enables a direct assessment of a model's ability to reason over any specialized knowledge. Since it is well-established that Wikipedia serves as a cornerstone of the training corpora for most models, PEB facilitates this with a Wikipedia collection of 960 puzzles of doctoral-level difficulty drawn from eight distinct disciplines. Our evaluation on PEB reveals a counter-intuitive phenomenon: most open-source reasoning models significantly underperform their standard, no-thinking counterparts. This finding uncovers a critical gap between proficiency in algorithmic reasoning and the application of reasoning to specialized knowledge domains. Thus, the construction of the PEB introduces a scalable method for rapidly generating diagnostic benchmarks in any specialized domain, enabling a robust evaluation of reasoning that is insensitive to prior data exposure.

TL;DR: We created a new benchmark by treating any learned specialized concept as a puzzle, revealing a critical gap between a model's algorithmic reasoning skill and its ability to reason about what it actually knows.