UCFE: A User-Centric Financial Expertise Benchmark for Large Language
  Models
Authors: name: Yuzhe Yang; name: Yifei Zhang; name: Yan Hu; name: Yilin Guo; name: Ruoli Gan; name: Yueru He; name: Mingcong Lei; name: Xiao Zhang; name: Haining Wang; name: Qianqian Xie; name: Jimin Huang; name: Honghai Yu; name: Benyou Wang
Published: 2024-10-17 22:03:52

Abstract: This paper introduces the UCFE: User-Centric Financial Expertise benchmark,
an innovative framework designed to evaluate the ability of large language
models (LLMs) to handle complex real-world financial tasks. UCFE benchmark
adopts a hybrid approach that combines human expert evaluations with dynamic,
task-specific interactions to simulate the complexities of evolving financial
scenarios. Firstly, we conducted a user study involving 804 participants,
collecting their feedback on financial tasks. Secondly, based on this feedback,
we created our dataset that encompasses a wide range of user intents and
interactions. This dataset serves as the foundation for benchmarking 12 LLM
services using the LLM-as-Judge methodology. Our results show a significant
alignment between benchmark scores and human preferences, with a Pearson
correlation coefficient of 0.78, confirming the effectiveness of the UCFE
dataset and our evaluation approach. UCFE benchmark not only reveals the
potential of LLMs in the financial sector but also provides a robust framework
for assessing their performance and user satisfaction.The benchmark dataset and
evaluation code are available.