Soundwave: Less is More for Speech-Text Alignment in LLMs
Authors: name: Yuhao Zhang; name: Zhiheng Liu; name: Fan Bu; name: Ruiyu Zhang; name: Benyou Wang; name: Haizhou Li
Published: 2025-02-18 14:36:39

Abstract: Existing end-to-end speech large language models (LLMs) usually rely on
large-scale annotated data for training, while data-efficient training has not
been discussed in depth. We focus on two fundamental problems between speech
and text: the representation space gap and sequence length inconsistency. We
propose Soundwave, which utilizes an efficient training strategy and a novel
architecture to address these issues. Results show that Soundwave outperforms
the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks,
using only one-fiftieth of the training data. Further analysis shows that
Soundwave still retains its intelligence during conversation. The project is
available at https://github.com/FreedomIntelligence/Soundwave.