FinAudio: A Benchmark for Audio Large Language Models in Financial
  Applications
Authors: name: Yupeng Cao, name: Haohang Li, name: Yangyang Yu, name: Shashidhar Reddy Javaji, name: Yueru He, name: Jimin Huang, name: Zining Zhu, name: Qianqian Xie, name: Xiao-yang Liu, name: Koduvayur Subbalakshmi, name: Meikang Qiu, name: Sophia Ananiadou, name: Jian-Yun Nie
Organization: The Fin AI

Audio Large Language Models (AudioLLMs) have received widespread attention
and have significantly improved performance on audio tasks such as
conversation, audio understanding, and automatic speech recognition (ASR).
Despite these advancements, there is an absence of a benchmark for assessing
AudioLLMs in financial scenarios, where audio data, such as earnings conference
calls and CEO speeches, are crucial resources for financial analysis and
investment decisions. In this paper, we introduce FinAudio, the first
benchmark designed to evaluate the capacity of AudioLLMs in the financial
domain. We first define three tasks based on the unique characteristics of the
financial domain: 1) ASR for short financial audio, 2) ASR for long financial
audio, and 3) summarization of long financial audio. Then, we curate two short
and two long audio datasets, respectively, and develop a novel dataset for
financial audio summarization, comprising the FinAudio benchmark.
Then, we evaluate seven prevalent AudioLLMs on FinAudio. Our
evaluation reveals the limitations of existing AudioLLMs in the financial
domain and offers insights for improving AudioLLMs. All datasets and codes will
be released.