Scaling Transformers for Low-Bitrate High-Quality Speech Coding
Authors: name: Julian D Parker; name: Anton Smirnov; name: Jordi Pons; name: CJ Carr; name: Zack Zukowski; name: Zach Evans; name: Xubo Liu
Published: 2024-11-29 16:58:02

Abstract: The tokenization of speech with neural audio codec models is a vital part of
modern AI pipelines for the generation or understanding of speech, alone or in
a multimodal context. Traditionally such tokenization models have concentrated
on low parameter-count architectures using only components with strong
inductive biases. In this work we show that by scaling a transformer
architecture with large parameter count to this problem, and applying a
flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to
reach state-of-the-art speech quality at extremely low bit-rates of 400 or
700 bits-per-second. The trained models strongly out-perform existing
baselines in both objective and subjective tests.