OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and
  Cleaning
Authors: name: Yifan Peng, name: Shakeel Muhammad, name: Yui Sudo, name: William Chen, name: Jinchuan Tian, name: Chyi-Jiunn Lin, name: Shinji Watanabe

The Open Whisper-style Speech Models (OWSM) project has developed a series of
fully open speech foundation models using academic-scale resources, but their
training data remains insufficient. This work enhances OWSM by integrating
YODAS, a large-scale web-crawled dataset with a Creative Commons license.
However, incorporating YODAS is nontrivial due to its wild nature, which
introduces challenges such as incorrect language labels and audio-text
misalignments. To address this, we develop a scalable data-cleaning pipeline
using public toolkits, yielding a dataset with 166,000 hours of speech across
75 languages. Our new series of OWSM v4 models, trained on this curated dataset
alongside existing OWSM data, significantly outperform previous versions on
multilingual benchmarks. Our models even match or surpass frontier industrial
models like Whisper and MMS in multiple scenarios. We will publicly release the
cleaned YODAS data, pre-trained models, and all associated scripts via the
ESPnet toolkit.