Advancing End-to-End Spoken Language Understanding with the Power of Large Language Models
Authors: Xuxin Cheng, Zhihong Zhu, Zhanpeng Chen, Xianwei Zhuang, Zhiqi Huang, Yuexian Zou


Reviewer paper summaries: The paper presents a new method SL3U for improving Spoken Language Understanding (SLU) using large language models. They combine the Whisper acoustic encoder with the LLaMA 2 language model and use an adapter module to bridge audio and text. This approach achieves SOTA performance on SLURP and MINDS-14 datasets. The training involves two steps: tuning the adapter to align audio and text models, and fine-tuning. Results show their method outperforms previous methods and even does better than ChatGPT in some tasks, demonstrating the effectiveness of using large language models in SLU.
Summary of strengths: - The combination of Whisper and LLaMA 2 using adapters is novel in NLP/ASR community, and leads to better SLU performance.
- The two-step training strategy us useful as the first step aligns speech and text well, boosting performance.
- Achieves top results on SLURP and MINDS-14, outperforming ChatGPT in some tasks.
- Superior performance of SL3U compared with other AudioLLMs such as SpeechGPT, LTU, LLaSM shows the significance of building E2E SLU systems.
Summary of weaknesses: - The authors conduct evaluation only on SLURP and MINDS-14 datasets. More variery would help. Some datasets can be ATIS, SNIPS, TIMIT etc.
- Some sections could be clearer, with more detailed explanations and step-by-step descriptions. For e.g. clarifying the flow of data from audio input through to final prediction would be beneficial.
- Limited comparisons with some popular E2E SLU methods - LAS and DeepSpeech2.
- Extensive fine-tuning may lead to overfitting. Further analysis on generalizability to unseen data would be beneficial.
Comments, suggestions and typos: **Comments/Suggestions**
- Test the model on more SLU datasets to show it works well in different areas.
- Make the method section easier to understand with detailed steps and visual aids.
- Compare the model with E2E SLU methods (LAS, DeepSpeed2) to highlight improvements.
- Analyze the risk of overfitting and discuss ways to ensure it works well with new data.

**Typos**
- In the introduction, change "subtasks, including intent detection and slot filling" to "subtasks: intent detection and slot filling".
Overall assessment scores: 3