Mind the Gap! Static and Interactive Evaluations of Large Audio Models
Authors: name: Minzhi Li; name: William Barr Held; name: Michael J Ryan; name: Kunat Pipatanakul; name: Potsawee Manakul; name: Hao Zhu; name: Diyi Yang
Published: 2025-02-21 20:29:02

Abstract: As AI chatbots become ubiquitous, voice interaction presents a compelling way
to enable rapid, high-bandwidth communication for both semantic and social
signals. This has driven research into Large Audio Models (LAMs) to power
voice-native experiences. However, aligning LAM development with user goals
requires a clear understanding of user needs and preferences to establish
reliable progress metrics. This study addresses these challenges by introducing
an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions
from 484 participants. Through topic modeling of user queries, we identify
primary use cases for audio interfaces. We then analyze user preference
rankings and qualitative feedback to determine which models best align with
user needs. Finally, we evaluate how static benchmarks predict interactive
performance - our analysis reveals no individual benchmark strongly correlates
with interactive results (tau leq 0.33 for all benchmarks). While combining
multiple coarse-grained features yields modest predictive power (R^2=0.30),
only two out of twenty datasets on spoken question answering and age prediction
show significantly positive correlations. This suggests a clear need to develop
LAM evaluations that better correlate with user preferences.