Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection
Authors: name: Zhiwei Liu, name: Yupen Cao, name: Yuechen Jiang, name: Mohsinul Kabir, name: Polydoros Giannouris, name: Chen Xu, name: Ziyang Xu, name: Tianlei Zhu, name: Tariquzzaman Faisal, name: Triantafillos Papadopoulos, name: Yan Wang, name: Lingfei Qian, name: Xueqing Peng, name: Zhuohan Xie, name: Ye Yuan, name: Saeed Almheiri, name: Abdulrazzaq Alnajjar, name: Mingbin Chen, name: Harry Stuart, name: Paul Thompson, name: Prayag Tiwari, name: Alejandro Lopez-Lira, name: Xue Liu, name: Jimin Huang, name: Sophia Ananiadou
Organization: The Fin AI

Large language models (LLMs) have been widely applied across various domains of finance. Since their training data are largely derived from human-authored corpora, LLMs may inherit a range of human biases. Behavioral biases can lead to instability and uncertainty in decision-making, particularly when processing financial information. However, existing research on LLM bias has mainly focused on direct questioning or simplified, general-purpose settings, with limited consideration of the complex real-world financial environments and high-risk, context-sensitive, multilingual financial misinformation detection tasks (\mfmd). In this work, we propose \mfmdscen, a comprehensive benchmark for evaluating behavioral biases of LLMs in \mfmd across diverse economic scenarios. In collaboration with financial experts, we construct three types of complex financial scenarios: (i) role- and personality-based, (ii) role- and region-based, and (iii) role-based scenarios incorporating ethnicity and religious beliefs. We further develop a multilingual financial misinformation dataset covering English, Chinese, Greek, and Bengali. By integrating these scenarios with misinformation claims, \mfmdscen enables a systematic evaluation of 22 mainstream LLMs. Our findings reveal that pronounced behavioral biases persist across both commercial and open-source models. This project will be available at https://github.com/lzw108/FMD.