Roadmap towards Superhuman Speech Understanding using Large Language
  Models
Authors: name: Fan Bu; name: Yuhao Zhang; name: Xidong Wang; name: Benyou Wang; name: Qun Liu; name: Haizhou Li
Published: 2024-10-17 06:44:06

Abstract: The success of large language models (LLMs) has prompted efforts to integrate
speech and audio data, aiming to create general foundation models capable of
processing both textual and non-textual inputs. Recent advances, such as
GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves
non-semantic information and world knowledge for deeper speech understanding.
To guide the development of speech LLMs, we propose a five-level roadmap,
ranging from basic automatic speech recognition (ASR) to advanced superhuman
models capable of integrating non-semantic information with abstract acoustic
knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark,
that standardizes critical aspects across various tasks in these five levels,
uncovering challenges in using abstract acoustic knowledge and completeness of
capability. Our findings reveal gaps in handling paralinguistic cues and
abstract acoustic knowledge, and we offer future directions. This paper
outlines a roadmap for advancing speech LLMs, introduces a benchmark for
evaluation, and provides key insights into their current limitations and
potential.