TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images
  Reasoning
Authors: name: Ming Li; name: Jike Zhong; name: Shitian Zhao; name: Haoquan Zhang; name: Shaoheng Lin; name: Yuxiang Lai; name: Wei Chen; name: Konstantinos Psounis; name: Kaipeng Zhang
Published: 2025-11-03 18:40:17

Abstract: The frontier of visual reasoning is shifting toward models like OpenAI o3,
which can intelligently create and operate tools to transform images for
problem-solving, also known as thinking-with-images in
chain-of-thought. Yet existing benchmarks fail to fully capture this advanced
capability. Even Visual Search, the most common benchmark for current
thinking-with-images methods, tests only basic operations such as
localization and cropping, offering little insight into more complex, dynamic,
and tool-dependent reasoning. We introduce TIR-Bench, a comprehensive
benchmark for evaluating agentic thinking-with-images across 13 diverse tasks,
each requiring novel tool use for image processing and manipulation in
chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from
leading open-sourced and proprietary models to those with explicit tool-use
augmentation. Results show that TIR-Bench is universally challenging, and
strong performance requires genuine thinking-with-images capabilities. Finally,
we present a pilot study comparing direct versus agentic fine-tuning.