Dissecting Tool-Integrated Reasoning: An Empirical Study and Analysis
Authors: name: Yufeng Zhao, name: Junnan Liu, name: Hongwei Liu, name: Dongsheng Zhu, name: Yuan Shen, name: Songyang Zhang, name: Kai Chen

Large Language Models (LLMs) have made significant strides in reasoning tasks
through methods like chain-of-thought (CoT) reasoning. However, they often fall
short in tasks requiring precise computations. Tool-Integrated Reasoning (TIR)
has emerged as a solution by incorporating external tools into the reasoning
process. Nevertheless, the generalization of TIR in improving the reasoning
ability of LLM is still unclear. Additionally, whether TIR has improved the
model's reasoning behavior and helped the model think remains to be studied. We
introduce ReasonZoo, a comprehensive benchmark encompassing nine diverse
reasoning categories, to evaluate the effectiveness of TIR across various
domains. Additionally, we propose two novel metrics, Performance-Aware Cost
(PAC) and Area Under the Performance-Cost Curve (AUC-PCC), to assess reasoning
efficiency. Our empirical evaluation demonstrates that TIR-enabled models
consistently outperform their non-TIR counterparts in both mathematical and
non-mathematical tasks. Furthermore, TIR enhances reasoning efficiency, as
evidenced by improved PAC and AUC-PCC, indicating reduced overthinking and more
streamlined reasoning. These findings underscore the domain-general benefits of
TIR and its potential to advance LLM capabilities in complex reasoning tasks.