Soundwave: Less is More for Speech-Text Alignment in LLMs
Authors: name: Yuhao Zhang, name: Zhiheng Liu, name: Fan Bu, name: Ruiyu Zhang, name: Benyou Wang, name: Haizhou Li

Existing end-to-end speech large language models (LLMs) usually rely on
large-scale annotated data for training, while data-efficient training has not
been discussed in depth. We focus on two fundamental problems between speech
and text: the representation space gap and sequence length inconsistency. We
propose Soundwave, which utilizes an efficient training strategy and a novel
architecture to address these issues. Results show that Soundwave outperforms
the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks,
using only one-fiftieth of the training data. Further analysis shows that
Soundwave still retains its intelligence during conversation. The project is
available at https://github.com/FreedomIntelligence/Soundwave.