SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction
  Fine-Tuning
Authors: name: Prabhat Pandey; name: Rupak Vignesh Swaminathan; name: K V Vijay Girish; name: Arunasish Sen; name: Jian Xie; name: Grant P. Strimel; name: Andreas Schwarz
Published: 2025-04-12 04:45:48

Abstract: We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset
designed for instruction fine-tuning and pre-training of speech-text large
language models (LLMs). SIFT-50M is built from publicly available speech
corpora, which collectively contain 14K hours of speech, and leverages LLMs
along with off-the-shelf expert models. The dataset spans five languages,
encompassing a diverse range of speech understanding as well as controllable
speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which
outperforms existing speech-text LLMs on instruction-following benchmarks while
achieving competitive performance on foundational speech tasks. To support
further research, we also introduce EvalSIFT, a benchmark dataset specifically
designed to evaluate the instruction-following capabilities of speech-text
LLMs.