SpeechEval: A Universal Benchmark for Speech-Text Foundation Models
Authors: Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, LIU ZHUOHAN, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, Nancy F. Chen


Reviewer paper summaries: Despite recent advancements in foundation models for multimodality, resources specifically designed to evaluate speech-to-text generation are limited. To fill this gap, the authors propose SpeechEval, a comprehensive benchmark for assessing the capabilities of speech-text foundation models(STFMs). By collecting existing resources and expanding new ones, SpeechEval includes 26 datasets across 8 tasks. These resources were created through manual efforts and automatic generation using, for example, large language models (LLMs). The primary evaluation involves assessing three recent STFM. Experimental results provide insightful findings, such as the sensitivity of models to prompt instructions and the need for further improvements to excel in SpeechEval. The analysis also shows that, for instance, open-source models like Llama3 can perform well as model-as-judge. Finally, the paper outlines several challenges for future research in this direction.
Summary of strengths: 1. SpeechEval appears to be one of the first benchmarks specifically designed to evaluate STFMs across a diverse range of tasks, including some new ones. Data resources and standardized evaluation protocols are always valuable and play a key role in guiding future research and development in the right direction.
2. The experiments evaluate three recent models on the newly proposed dataset, which, along with the evaluation process, can serve as valuable references for future work in evaluating and developing better systems. The analysis is commendable, as it supports the selection of prompts and the use of open-source LLMs in the evaluation.
3. It appears that the new SpeechEval comprehensively covers both existing and newly created resources, reflecting significant investment in cost, time, and manual effort. It is commendable that the evaluation goes beyond merely reporting numbers, offering valuable insights and identifying challenges for future research.
Summary of weaknesses: 1. This work is mainly evaluating existing models using the new benchmark without introducing significant innovations in methods, theory, or groundbreaking findings. Although the proposed resources are valuable and substantial efforts are evident, the contributions in their current form may not be sufficient for a top-tier conference.
2. The analysis results indicate that Llama3 performs well in the evaluation and is better than closed-source models like GPT-4. However, the paper also mentions "ongoing challenges in selecting appropriate models to ensure both accessibility and robustness," and it remains unclear how Llama3 addresses these challenges. LLMs often suffer from many issues, such as "hallucinations," where they generate meaningless or incorrect responses no matter whether open-source or closed-source. This raises concerns about, for example, how to parse such responses when using LLMs for evaluation purposes. This is just one example of the potential issues. Most would prefer an evaluation process that is automatic, consistent, and reproducible, without relying on manual intervention.
3. Despite additional details in the appendix, the presentation of the work requires refinement, and some statements are unclear, making it difficult to understand the work accurately. Please refer to the comments below for specific suggestions.
Comments, suggestions and typos: The presentation of this work requires a double-check and major revision. What is listed here are just some examples. 
+ Line 054 - It is not clear why "given that these models follow instructions, traditional evaluation metrics can not fully meet the requirements and could introduce bias in evaluation" without any reference provided.
+ Line 117, 131, 150 - Text or vision-based benchmarks are not closely related to this work. Existing text-speech evaluation methods and resources are not fully discussed to distinguish and motivate this work.
+ Line 225 - What are these LLMs evaluated for?
+ Line 313 - What are these LLMs involved in the data generation?
+ Line 410 - It might be better to refer to a table for listing prompts instead of repeating prompts in text.
+ Figure 2 - What does the y-axis mean?
+ Section 4 - It is recommended to font load the main results first and then discuss the options for prompt tempaltes and evaluation LLMs used in the analyses later.
Overall assessment scores: 2.5
Metareview score: 3
Metareview: This work presents SpeechEval, a benchmark to evaluate Speech-Text Foundation Models (STFMs). The benchmark consists of a total of 8 distinct tasks and 26 datasets, 7 of which are newly curated (derived from existing datasets). 

The 8 tasks are: Automatic Speech Recognition (ASR), Speech Question Answering (SQA), Audio Question Answering (AQA), Emotion Recognition (ER), Speech Instruction (SI), Audio Captioning (AC), Accent Recognition (AR) and Gender Recognition (GR). Tasks like AC and AQA go beyond comprehending spoken content and demand audio scene understanding.

The paper evaluate 3 SFTMs: QWEN-audio, SALMONN and WavLLM, as well as a cascaded baseline of Whisper-v3 and LLama3.

Additionally, the paper sheds some light on the critical challenge of prompt robustness when evaluating speech-text LLMs.  To minimize the model’s brittleness, the authors incorporate diverse prompt templates into SpeechEval for some tasks

The authors also rely on model-as-judge as the main metric for this benchmark. Two text-only LLMs were compared for their performance as judges: GPT-4 and LLama3.  

This work provides a leaderboard with open-sourced code & data.
Summary of reasons to publish: - A commendable effort providing a useful benchmark that goes beyond existing benchmarks like SUPERB to evaluate instruction-following speech-text LLMs.
- Source code and datasets will be made available for the community with a leaderboard comparing state-of-the-art models on a set of challenging tasks that cover a wide range of specifications. Benchmark construction is definitely an important aspect of scientific contribution.