TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and
  Novel Tic-Tac-Toe-style Games
Authors: name: Prakamya Mishra, name: Jiang Liu, name: Jialian Wu, name: Xiaodong Yu, name: Zicheng Liu, name: Emad Barsoum
Organization: AMD

Large reasoning models (LRMs) have demonstrated impressive reasoning
capabilities across a broad range of tasks including Olympiad-level
mathematical problems, indicating evidence of their complex reasoning
abilities. While many reasoning benchmarks focus on the STEM domain, the
ability of LRMs to reason correctly in broader task domains remains
underexplored. In this work, we introduce TTT-Bench, a new benchmark
that is designed to evaluate basic strategic, spatial, and logical reasoning
abilities in LRMs through a suite of four two-player Tic-Tac-Toe-style games
that humans can effortlessly solve from a young age. We propose a simple yet
scalable programmatic approach for generating verifiable two-player game
problems for TTT-Bench. Although these games are trivial for humans, they
require reasoning about the intentions of the opponent, as well as the game
board's spatial configurations, to ensure a win. We evaluate a diverse set of
state-of-the-art LRMs, and discover that the models that excel at hard
math problems frequently fail at these simple reasoning games. Further testing
reveals that our evaluated reasoning models score on average downarrow 41\%
\& downarrow 5\% lower on TTT-Bench compared to MATH 500 \& AIME 2024
respectively, with larger models achieving higher performance using shorter
reasoning traces, where most of the models struggle on long-term strategic
reasoning situations on simple and new TTT-Bench tasks.