FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial
  Reasoning
Authors: name: Zhuohan Xie, name: Dhruv Sahnan, name: Debopriyo Banerjee, name: Georgi Georgiev, name: Rushil Thareja, name: Hachem Madmoun, name: Jinyan Su, name: Aaryamonvikram Singh, name: Yuxia Wang, name: Rui Xing, name: Fajri Koto, name: Haonan Li, name: Ivan Koychev, name: Tanmoy Chakraborty, name: Salem Lahlou, name: Veselin Stoyanov, name: Preslav Nakov

Multi-step symbolic reasoning is critical for advancing downstream
performance on financial tasks. Yet, benchmarks for systematically evaluating
this capability are lacking. Existing datasets like FinQA and ConvFinQA
supervise only final numerical answers, without assessing intermediate
reasoning steps. To address this, we introduce FinChain, the first symbolic
benchmark designed for verifiable Chain-of- Thought (CoT) financial reasoning.
Spanning 54 topics across 12 financial domains, Fin- Chain offers five
parameterized templates per topic, each varying in reasoning complexity and
domain expertise required. Each dataset instance includes an executable Python
trace, enabling automatic generation of extensive training data and easy
adaptation to other domains. We also introduce ChainEval, a new metric for
automatic evaluation of both final answers and intermediate reasoning.
Benchmarking 30 LLMs on our dataset, we find that even state-of-the-art models
have considerable room for improvement in multi-step financial reasoning. All
templates and evaluation metrics for FinChain are available at https:
//github.com/mbzuai-nlp/finchain.