Title: Chunk Based Speech Pre-training with High Resolution Finite Scalar Quantization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised Learning, Finite Scalar Quantization, Speech Recognition, Speech Translation

Abstract: Low latency speech human-machine communication is becoming increasingly necessary as speech technology advances quickly in the last decade.  One of the primary factors behind the advancement of speech technology is self-supervised learning. Most self-supervised learning algorithms are designed with full utterance assumption and compromises have to made if partial utterances are presented, which are common in the streaming applications.  In this work, we propose a chunk based self-supervised learning (Chunk SSL) algorithm as an unified  solution for both streaming and offline speech pre-training.  Chunk SSL is optimized with the masked prediction loss and 
an acoustic encoder is encouraged to restore indices of those masked speech frames with help from unmasked frames in the same chunk and preceding chunks.  A copy and append data augmentation approach is proposed to conduct efficient chunk based pre-training.
Chunk SSL utilizes a finite scalar quantization (FSQ) module to discretize input speech features and our study shows a high resolution FSQ codebook, i.e., a codebook with vocabulary size up to a few millions, is beneficial to transfer knowledge from the pre-training task to the downstream tasks.   A group masked prediction loss is employed during pre-training to alleviate the high memory and computation cost introduced by the large codebook. The proposed approach is examined in two speech to text tasks, i.e., speech recognition and speech translation. Experimental results on the \textsc{Librispeech} and \textsc{Must-C} datasets show that the proposed method could achieve
very competitive results for speech to text tasks at both streaming and offline modes.

TL;DR: This work proposes a chunkwise self-supervised training based on high resolution finite scalar quantization and achieve competitive results for speech to text tasks.