Chunk Based Speech Pre-training with High Resolution Finite Scalar Quantization

## Abstract

Abstract Low latency speech human-machine communication is becoming increasingly necessary as speech technology advances quickly in the last decade.
One of the primary factors behind the advancement of speech technology is self-supervised learning.
Most self-supervised learning algorithms are designed with full utterance assumption and compromises have to made if partial utterances are presented, which are common in the streaming applications.
In this work, we propose a chunk based self-supervised learning (Chunk SSL) algorithm as an unified solution for both streaming and offline speech pre-training.
Chunk SSL is optimized with the masked prediction loss and
an acoustic encoder is encouraged to restore indices of those masked speech frames with help from unmasked frames in the same chunk and preceding chunks.
A copy and append data augmentation approach is proposed to conduct efficient chunk based pre-training.
Chunk SSL utilizes a finite scalar quantization (FSQ) module to discretize input speech features and our study shows a high resolution FSQ codebook, i.e., a codebook with vocabulary size up to a few millions, is beneficial to transfer knowledge from the pre-training task to the downstream tasks. A group masked prediction loss is employed during pre-training to alleviate the high memory and computation cost introduced by the large codebook.
The proposed approach is examined in two speech to text tasks, i.e., speech recognition and speech translation.
Experimental results on the Librispeech and Must-C datasets show that the proposed method could achieve
very competitive results for speech to text tasks at both streaming and offline modes.

## 1 Introduction

Transcribe speech into text in real time is critical for applications requiring immediate feedback, such as real-time transcription of broadcast contents, voice assistants, and simultaneous speech translation etc.
It requires processing audio incrementally as it is received, rather than waiting for the entire utterance to be available.
For neural network based end-to-end systems,
the requirement includes two-fold. First, a speech encoder, such as causal encoder (Zhang et al., 2020b) and chunk encoder (Tsunoo et al., 2019; Shi et al., 2021), is able to process input audio cumulatively without dependency on the future input.
Second, a decoder is required to decode transcription incrementally based on partial encoder outputs.
Frame-Synchronous decoders (Dong et al., 2020), such as CTC (Graves et al., 2013) Transducer (Graves, 2012) CIF (Dong & Xu, 2020) and TAED (Tang et al., 2023), are capable of generating transcription based on partial encoder outputs and meet the streaming requirement naturally.

Self-supervised pre-training leverages vast amounts of unlabeled data and learn universal feature representations for downstream tasks, especially for tasks with limited supervised training data.
Speech pre-training methods have emerged as the backbone of many speech processing tasks (Chung et al., 2018; Baevski et al., 2020b; Hsu et al., 2021b; Chen et al., 2021; Chiu et al., 2022; Tang et al., 2022; Baade et al., 2025).
Those methods are designed for the
speech applications with full utterance available, and compromises have to be made if the downstream tasks are streaming applications.
Chiu et al. (2022) propose to conduct pre-training with a causal encoder, which sacrifices right context information. Fu et al. (2024) modify the encoder structure
and conduct continual pre-training to adapt the model for streaming applications.

Figure: Figure 1: Chunkwise self-supervised training.
Refer to caption: figures/Chunk_SSL.drawio.png

In all those methods aforementioned, a dedicated pre-trained model has to be built for the streaming scenario instead of sharing the same model with the offline application.

In this study, we focus on building an encoder suitable for both streaming and offline modes,
with a chunkwise self-supervised learning (Chunk SSL) framework as depicted in [Figure 1](https://arxiv.org/html/2509.15579v1#S1.F1).
Chunk SSL aims to
restore discrete indices of the masked frames based on unmasked frames in the last chunk and previous chunks.
The discrete indices are estimated with finite scalar quantization (Mentzer et al., 2024) (FSQ) with vocabulary size up to millions ([Section 3](https://arxiv.org/html/2509.15579v1#S3)).The pre-training FSQ token has more fine-grained resolution compared with the downstream modeling units, such as phonemes or sentencepiece tokens,
which usually are with vocabulary size ranging from tens to tens of thousands.
We hypothesize that speech frames associated with a high resolution FSQ token are mainly mapped to an unique modeling unit in the downstream task and it makes the knowledge transfer easier from the pre-training stage to the fine-tuning stage.
However, those high resolution FSQ tokens pose a great challenge for modeling and we propose to decompose a large codebook
into small channel based sub-codebooks to alleviate the memory and computation cost during pre-training. Details could be found in [Section 3](https://arxiv.org/html/2509.15579v1#S3).

In [Figure 1](https://arxiv.org/html/2509.15579v1#S1.F1), speech features are first extracted from the input audio, and grouped into equal sized chunks.
Instead of calculating those chunks from left to right sequentially, a copy and append data augmentation (CADA) is introduced to parallelize the computation for all chunks in the same utterance ([Section 2](https://arxiv.org/html/2509.15579v1#S2)).
CADA augments the input sequence by copying input chunks and appending them to the end of utterance as extended chunks.
Masking is applied to frames in extended chunks only.
Augmented utterances are then processed by a CADA compatible Conformer encoder, which could handle those augmented chunks properly even though frames in extended chunks are with altered location information. Finally,
the model is optimized by restoring FSQ indices of those masked frames from Conformer encoder outputs.
Experiments on Librispeech and MuST-C datasets indicate that
the same model initialized with Chunk SSL could achieve very competitive results
on both streaming and offline speech to text tasks.
It shows that Chunk SSL eliminates the need to build a dedicated streaming and dedicated offline model.
To summarize, our contributions includes:

- 1.
We propose a chunkwise speech self-supervised learning algorithm for both streaming and offline speech to text tasks.
- 2.
A copy and append data augmentation improves the pre-training efficiency by reusing the chunk level computation results and parallelizing the chunkwise based computation.
- 3.
FSQ is employed to generate high resolution speech codebook and a group masked prediction loss is proposed to alleviate the computation challenge.
- 4.
Our results show the proposed method can build one model for both scenarios and achieve competitive results on the Librispeech and MuST-C datasets.

## 2 Copy and Append Data Augmentation

The naive Chunk SSL algorithm, which recovers masked frame indices in the right most chunk based on the unmasked frames in the same chunk and preceding chunks, is executed chunk by chunk in a sequential order instead of computing all chunks from one utterance in a parallel fashion.
Inspired by the implementation of the chunk encoder with a look-ahead chunk for the streaming speech translation (Liu et al., 2021),
we propose a copy and append data augmentation for the Chunk SSL,
which reorganizes
input data and changes the augmented data computation, but it is still strictly equivalent to the computation in the naive Chunk SSL algorithm.

The pre-training procedure is described in [Figure 1](https://arxiv.org/html/2509.15579v1#S1.F1). Input features are first downsampled with a stacked subsampling module. The outputs are then divided into chunks with fixed duration, They are considered as base chunks.
The consecutive chunk of each base chunk is copied and appended to the end of the utterance in a sequential order. Those newly created chunks are called extended chunks.
For example, there are 12 speech input frames in [Figure 1](https://arxiv.org/html/2509.15579v1#S1.F1) after subsampling. Assuming the chunk size is 4 frames, the speech features are segmented into three base chunks (in cyan): “B1”, “B2” and “B3”.
Chunk “B2” and “B3” are copied and appended to the end of the utterance as “E1” and “E2” (in lime).
They correspond to the extended chunks of “B1” and “B2” respectively.
There is no extended chunk for the last chunk (“B3”).
Extended chunks act as right most chunks with different preceding chunks in the naive Chunk SSL algorithm and
masking is only applied on frames of extended chunks.

A speech encoder, such as Transformer (Vaswani et al., 2017) and Conformer (Gulati et al., 2020), contains two types of modules. One is the intra-frame module where the computation is based on one frame and no location information or other frames are required, for example, LayerNorm and feedforward layers.
The other type is the inter-frame module where the computation is related to the input location and requires information from neighbouring frames.
The inter-frame modules include self-attention and convolutional layers.
An augmented utterance aforementioned, can be processed as a normal utterance in the intra-frame modules and no modification is required.
On the other hand, the position information is invalid among extended chunk frames, since they are copied and appended to the end of utterance.
Modifications are introduced for two inter-frame modules: self-attention and convolutional layers, to handle inconsistent position information in augmented utterances as described in
the following subsections.

### 2.1 CADA self-attention module

The self-attention module exchanges information within the same sequence according to the attention weights. Assume ${\bm{X}}=\{{\bm{x}}_{0},{\bm{x}}_{1},\cdots,{\bm{x}}_{N-1}\}$ is a $N$ length input sequence
to a self-attention layer, the attention weights ${\bm{A}}=\{{A}_{i,j}\}$ are calculated as

$$ ${A}_{i,j}=\frac{\exp\bigl(\alpha_{i,j}\beta({\bm{W}}_{q}{\bm{x}}_{i})^{T}({\bm{W}}_{k}{\bm{x}}_{j})\bigr)}{\sum_{\tau}\exp\bigl(\alpha_{i,\tau}\beta({\bm{W}}_{q}{\bm{x}}_{i})^{T}({\bm{W}}_{k}{\bm{x}}_{\tau})\bigr)}$ (1) $$

where $\beta=\frac{1}{\sqrt{d}}$ is a scaling factor,
${\bm{W}}_{q},{\bm{W}}_{k}\in\mathcal{R}^{d\times d}$ are transform matrices, and $d$ is the dimension of model embedding(^1^11Single head attention is discussed for simplicity.).
$\bm{\alpha}=\{\alpha_{i,j}\},i,j\in[0,N-1]$ is a masking matrix with value 1 or 0 for each component and $\alpha_{i,\tau}=0$ means ${\bm{x}}_{\tau}$ will not contribute the computation of attention weights ${A}_{i,*}$ and the corresponding attention output of ${\bm{x}}_{i}$.
The self-attention module leverages the attention mask $\mathbf{\alpha}$ to control information exchange within frames from the same sequence.
For example, an encoder usually sets all $\alpha_{i,\tau}=1$ for full information access.

Assuming $N$ is the number of input frames in an utterance and is a multiple of chunk size $C$.
The augmented input sequence length is $N^{\prime}=2N-C$, the number of base chunks is $M=N/C$ and the number of extended chunks is $M-1$.
We define $m(i)=\lfloor{i/C}\rfloor$ as the chunk index of frame $i$. If the left context is infinite, i.e., the model could access all history information, a CADA masking matrix $\bm{\alpha}$ is defined as

$$ $\alpha_{i,j}=\begin{cases}1,&\text{if }m(i)>m(j)\text{ and }i<N\\ &\text{or if }m(i)==m(j)-M\\ &\text{or if }m(i)==m(j)\\ &\text{or if }m(i)\geq m(j)+M\\ 0,&\text{otherwise}\end{cases}$ (2) $$

where $i,j\in[0,N^{\prime}-1]$. The first two rows in [Eq. 2](https://arxiv.org/html/2509.15579v1#S2.E2) define the masking matrix for a frame $i$ in base
chunks and the fourth row is for frames in extended chunks only.
To be more specific,

- 1.
the first row means a frame $i$ from base chunks could access all
information in the previous chunks
- 2.
the second row represents a frame $i$ from base chunks can also access limited future information from its extended chunk
- 3.
the third row indicates a frame $i$ could access all frames within the same chunk
- 4.
the fourth row means a frame $i$ from extended chunks can access all information in its base chunk and preceding chunks of the base chunk
- 5.
there is no information exchange for all other cases as shown in the fifth row.

Figure: (a) Self-Attention CADA Masking for the augmented utterance with original sequence length 6 and chunk size 2.
Refer to caption: figures/Chunk_Convolution.drawio.png

[2(a)](https://arxiv.org/html/2509.15579v1#S2.F2.sf1) depicts a CADA masking example for an utterance with 6 frames before CADA. The first column is the frame index. The number in the bracket (brown) stands for the extended chunk frame’s original position, i.e., the position of the base chunk frame that the frame is copied from.
${\bm{\alpha}}$ for the extended chunks are presented in italic fonts. A self-attention module equipped with a CADA masking matrix can process a CADA sequence with position information preserved.

### 2.2 CADA convolutional module

A convolutional layer has a set of filters which slide across input features within the same utterance and extract localized representations. A convolutional layer assumes the input frames organized as consecutively, which no longer holds true for extended chunk frames in a CADA utterance.
This problem can be resolved via a chunk based convolutional computation (Li et al., 2023).
Without loss of generality, we assume the convolution module is a depth-wise convolution layer with $L_{lc}$ left and $L_{rc}$ right context frames. The modified computation includes three steps.
First, we separate the CADA utterance into chunks and pair a base chunk and its extended chunk together, for example “B1” and “E1” in [Figure 1](https://arxiv.org/html/2509.15579v1#S1.F1); then the last $L_{lc}$ frames from the chunk preceding the base chunk
are appended to the left as left context. If the base chunk is the first chunk, $L_{lc}$ zero padding frames are attached. Similarly, $L_{rc}$ zero padding frames are appended to the right and it leads to a new concatenated subsequence with length $L_{lc}+2C+L_{rc}$.
Second, the concatenated subsequence is fed to the convolution module and obtains output subsequence with length $2C$. $L_{lc}$ and $L_{rc}$ context frames are absorbed during convolution
computation. The first $C$ output frames correspond to the base chunk and the second $C$ frames are for the extended chunk.
Finally, those output chunks are placed back to their original positions in the augmented utterance as the outputs from the CADA convolutional module.
[2(b)](https://arxiv.org/html/2509.15579v1#S2.F2.sf2) demonstrates three steps to conduct convolution computation on an augmented utterance with base sequence length 12 and chunk size 4.

### 2.3 Dynamic chunk based pre-training

Dynamic chunk training (Zhang et al., 2020a; Weninger et al., 2022; Li et al., 2023) is a useful fine-tuning approach for the chunk based ASR encoder.
During training, the chunk duration varied from hundreds of million seconds to a few seconds is randomly chosen for every model update. A model built with dynamic chunk training is suitable for both streaming and offline modes.
We extend dynamic chunk training for the Chunk SSL pre-training. Chunk duration is chosen from 6 durations ($[640,1280,1920,2560,3200,3840]$ million seconds) randomly for every model update.

## 3 High Resolution Finite Scalar Quantization

### 3.1 Quantization module

As shown in [Figure 1](https://arxiv.org/html/2509.15579v1#S1.F1), we leverage finite scalar quantization (Mentzer et al., 2024) to quantize
input frames and obtain their discrete indices.
A FSQ module is built prior to the Chunk SSL pre-training.
In the quantization module, an utterance based channel mean and variance normalization is first applied to input frames ${\bm{X}}$ to obtain $\hat{{\bm{X}}}=\{\hat{{\bm{x}}}_{i}\}$, then it is processed by a FSQ encoder $\text{Enc}_{f}$ and projected into a low-dimension space with size $d^{\prime}$, where $d^{\prime}\ll d$. The output of each channel $r\in[1,d^{\prime}]$ for input frame $i$ is rounded to an integer $h_{i,r}$

$$ $h_{i,r}=\textsc{Round}\bigl(\lfloor{K_{r}/2}\rfloor\tanh(\textsc{Enc}_{f}(\hat{{\bm{x}}}_{i})[r]\bigr),\texttt{and}\;\;h_{i,r}\in[-\lfloor{K_{r}/2}\rfloor,\lfloor{K_{r}/2}\rfloor]$ (3) $$

where Round is a bounded round operation and $K_{r}$ is the corresponding number of levels at channel $r$.
Then the output ${\bm{h}}_{i}=\{h_{i,r}\}$ is projected back to the original space via a decoder $\tilde{{\bm{x}}}_{i}=\textsc{Dec}_{f}({\bm{h}}_{i})$. The FSQ module is optimized by matching reconstructed $\tilde{{\bm{x}}}_{i}$ and input $\hat{{\bm{x}}}_{i}$.

### 3.2 Group masked prediction loss

The acoustic encoder is optimized with masked prediction loss. We encourage the Chunk SSL model to reconstruct the masked frames based on the context information provided.
More specifically, we mask half consecutive frames in every extended chunk. The start position of the masking frame is randomly chosen from $[0,\frac{C}{4}]$ at every extended chunk, so we have enough unmasked frames on both sides of the masked frames as context to infer those masked frames.

Assuming the Chunk SSL model outputs for the augmented sequence is ${\bm{O}}=\{{\bm{o}}_{i}\}$. The FSQ index of $x_{i}$ is denoted as $\mu_{i}$ and
${\mathbf{e}}_{\mu_{i}}$ is the output embedding of $\mu_{i}$. The masked prediction loss is defined as

$$ $\mathcal{L}_{m}=-\sum_{i\in{\bm{M}}}\log\frac{\exp({\bm{o}}_{i}^{T}{\bm{e}}_{\mu_{i}})}{\sum_{j=1}^{V}\exp({\bm{o}}_{i}^{T}{\bm{e}}_{j})}$ (4) $$

where ${\bm{e}}_{j}$ is the $j$-th output embedding, ${\bm{M}}$ is the masked frames set, and $V$ is vocabulary size from the quantization module.

We hypothesize a high resolution FSQ codebook would be beneficial for downstreaming tasks due to two reasons. First, FSQ does not suffered from codebook collapse and can achieve high codebook usage.
Second, a high resolution codebook divides the feature space into more fine-grained bins and frames sharing the same token index might be more closer to each other compared with frames associated with a token index from a low resolution codebook.
When the FSQ codebook is large enough, each FSQ token could be mainly associated with one modeling unit, such as phoneme, in the downstreaming task, hence it will make the knowledge transfer easier from the pre-training stage to the fine-tuning stage. This hypothesis is examined in [Section 5.2](https://arxiv.org/html/2509.15579v1#S5.SS2).
However, a high resolution FSQ codebook poses a great challenge for optimization.
For example, the codebook with vocabulary size 1,265,625 and embedding size 512 would take about 2.4G memory if they are stored in float data type.
In order to alleviate this issue, we propose to decompose the codebook into a group of channel based sub-codebooks and compute prediction loss for
each group one by one. Given frame $i$, the sub-codebook index at $r$th channel is $h_{i,r}$.
We define the group masked prediction loss as

$$ $\mathcal{L}_{m}=-\sum_{i\in{\bm{M}}}\sum_{r}\log\frac{\exp({\bm{o}}_{i}^{T}{\bm{e}}^{r}_{h_{i,r}})}{\sum_{j=1}^{K_{r}}\exp({\bm{o}}_{i}^{T}{\bm{e}}^{r}_{j})}$ (5) $$

where ${\bm{e}}_{j}^{r}$ is the $j$th output embedding at the $r$th channel sub-codebook. Optimization on sub-codebook individually is equivalent to optimizing the full codebook, since a perfect system that accomplishes the
sub-tasks in [Eq. 5](https://arxiv.org/html/2509.15579v1#S3.E5) could solve the [Eq. 4](https://arxiv.org/html/2509.15579v1#S3.E4) too, but with much less memory requirement.
For the same codebook with number of levels per channel $[5,5,5,5,5,5,3,3,3,3]$ and size 1,265,625 aforementioned, they only take about 84k storage memory.

## 4 Experimental Settings

### 4.1 Data

Pre-training:
There are two pre-training tasks in this work, i.e., FSQ SSL training and Chunk SSL training. We first build FSQ module to discretize the input
speech feature and then pre-train speech encoder with Chunk SSL to generate contextual representation.
Libri-light (Kahn et al., 2019), which includes 60k hours of unlabelled English speech data, is used in both pre-training tasks.

Fine-Tuning: For speech recognition task, the pre-trained models are finetuned and evaluated on Librispeech (Panayotov et al., 2015) dataset. We use two dev sets for development and report all results from dev and test sets.
For speech translation task, experiments are conducted on two MuST-C (Gangi et al., 2019)
language pairs: English to German (EN$\rightarrow$DE) and English to Spanish (EN$\rightarrow$ES). Sequence level knowledge
distillation (Kim & Rush, 2016) is applied to boost the speech translation quality.
The models are developed on
the dev set, and the final results are reported on the tst-COMMON set.

### 4.2 Model configures

The speech encoder is a Conformer (Gulati et al., 2020) based chunk encoder.
The encoder is equipped with a relative positional embedding (Shaw et al., 2018) and starts with a stacking layer to down-sample the input features by four times.
Two model configurations: base and large, have been explored. The base encoder has 12 Conformer layers, input embedding size of 512, 8 attention heads, feedforward layer dimension 2048 and convolutional module kernel size 31. The large encoder has 24 Conformer layers, input embedding size of 768, 16 attention heads, feedforward layer dimension 3072 and convolutional module kernel size 5.
Transducer is adopted in the fine-tuning experiments.
The predictor module is with one Transformer layer (Vaswani et al., 2017) for speech recognition tasks and two layers for speech translation tasks.
The input embedding and feedforward layer dimension are the same as ones in the encoder setting if not mentioned specifically. The joint module is a feedforward layer as (Zhang et al., 2020b) with embedding dimension 1024.

Input speech is represented as 80-dimensional log mel-filterbank coefficients computed every 10ms with a 25ms window. Global channel mean and variance normalization is applied so the trained model could be used for both streaming and offline scenarios.
We set the maximum utterance duration to 75 seconds and minimum duration to 5 seconds in pre-training.
During fine-tuning,
a look-ahead chunk (Shi et al., 2021; Liu et al., 2021; Tang et al., 2023) is utilized and
dynamic chunk training is employed by default. We alternate between offline training with infinite chunk size, and streaming training with chunk duration sampled from [160, 320, 640, 960, 1280, 1600] ms randomly within each epoch.
More details about optimization, such as batch sizes and training schedulers, for different experiments are presented in [Appendix A](https://arxiv.org/html/2509.15579v1#A1).
The target labels are encoded with SentencePiece (Kudo & Richardson, 2018). For both speech recognition and speech translation tasks, the vocabulary is an unigram model with size 1024 and full character coverage on the corresponding training text data.

The final results are evaluated using an averaged model from checkpoints of the best 10 epochs.
Speech recognition experiments are evaluated with WER while speech translation results are measure with scacre BLEU score(^2^22signature: nrefs:1$|$case:mixed$|$eff:no$|$tok:13a$|$smooth:exp$|$version:2.4.2).
We report both streaming and offline results.
For the streaming decoding, we set the chunk size to 320ms by default if not mentioned specifically. No language model is used in all experiments.

### 4.3 FSQ module

The FSQ module is built with the ResNet based encoder and decoder (Langman et al., 2024). There are 12 layers for both encoder and encoder with embedding size 512.
The frontend processing is the same as the one described in [Section 4.2](https://arxiv.org/html/2509.15579v1#S4.SS2). The mel-filterbank feature is with a 25ms window and 10ms shift.
4 mel-filterbank stacked features are used as input for the FSQ encoder and the reconstructed target for the FSQ decoder. The default encoder output dimension is 12 with number of levels per channel $[5,5,5,5,5,3,3,3,3,3,3,3]$. It is equivalent to a codebook with vocabulary size 6,834,375.

## 5 Experimental Results

### 5.1 Main results

**Table 1: Offline WERs for models trained on the 960 hours LibriSpeech data (WER $\downarrow$).**
| Model | size(M) | dev-clean | dev-other | test-clean | test-other | ave. |
| --- | --- | --- | --- | --- | --- | --- |
| wav2vec2 base (Baevski et al., 2020b) | 95 | 3.2 | 8.9 | 3.4 | 8.5 | 6.0 |
| wav2vec2 large (Baevski et al., 2020b) | 317 | 2.1 | 4.5 | 2.2 | 4.5 | 3.3 |
| w2v-Conformer XL (Zhang et al., 2020c) | 600 | 1.7 | 3.5 | 1.7 | 3.5 | 2.6 |
| BEST-RQ (Chiu et al., 2022) | 600 | 1.5 | 2.8 | 1.6 | 2.9 | 2.2 |
| Chunk SSL Base | 79 | 2.0 | 5.0 | 2.1 | 4.9 | 3.5 |
| Chunk SSL Large | 337 | 1.8 | 4.2 | 1.9 | 4.2 | 3.0 |

**Table 2: Streaming WERs for models trained on the 960 hours LibriSpeech data (WER $\downarrow$).**
| Model | size(M) | dev-clean | dev-other | test-clean | test-other | ave. |
| --- | --- | --- | --- | --- | --- | --- |
| Conformer (Chiu et al., 2022) | 100 | 4.1 | 10.3 | 4.5 | 9.8 | 7.2 |
| Conformer (Chiu et al., 2022) | 600 | 3.9 | 9.8 | 4.4 | 9.4 | 6.9 |
| wav2vec2 (Chiu et al., 2022) | 600 | 2.7 | 8.0 | 2.9 | 7.9 | 5.4 |
| w2v-BERT (Chiu et al., 2022) | 600 | 2.7 | 8.4 | 3.0 | 8.1 | 5.6 |
| BEST-RQ (Chiu et al., 2022) | 600 | 2.5 | 6.9 | 2.8 | 6.6 | 4.7 |
| Chunk SSL Base | 79 | 2.3 | 6.6 | 2.5 | 6.4 | 4.5 |
| Chunk SSL Large | 337 | 2.1 | 5.5 | 2.3 | 5.5 | 3.9 |

**Table 3: Speech Translation Result on the MuST-C dataset (BLEU $\uparrow$).**
| Lang | Init. | Offline | Streaming |  |  |
| --- | --- | --- | --- | --- | --- |
| dev | tst-COMMON | dev | tst-COMMON |  |  |
| En$\rightarrow$Es | ASR | 28.3 | 24.3 | 24.9 | 21.4 |
| Chunk SSL | 30.3 | 26.3 | 26.8 | 23.3 |  |
| En$\rightarrow$De | ASR | 21.7 | 21.8 | 18.9 | 18.5 |
| Chunk SSL | 23.7 | 23.6 | 20.1 | 20.2 |  |

The pre-trained Chunk SSL models are fine-tuned with dynamic chunk training and they could be used for both streaming and offline ASR.
We list the offline and streaming recognition results from the Chunk SSL models in [Table 1](https://arxiv.org/html/2509.15579v1#S5.T1) and [Table 2](https://arxiv.org/html/2509.15579v1#S5.T2).
The reported literature results are presented in the first part of the tables.
It is clear that the chunk SSL pre-trained models could achieve very competitive results compared to other strong baselines.
The base and large models outperform the corresponding wav2vec2 base and large models.

In the streaming evaluation, the Chunk SSL model with the base configuration excels all models reported in the literature shown in [Table 2](https://arxiv.org/html/2509.15579v1#S5.T2). The large configuration model pushes the WER even lower and it achieves another 0.5 average WER reduction compared to the base configure model. Comparing results in [Table 1](https://arxiv.org/html/2509.15579v1#S5.T1) and [Table 2](https://arxiv.org/html/2509.15579v1#S5.T2),
the proposed method significantly reduces the performance gap between the streaming and offline applications. The WER gap between the large configure
models is 0.8 in average for Chunk SSL while the average WER difference is 2.5 for BEST-RQ.
Note, literature models listed in [Table 1](https://arxiv.org/html/2509.15579v1#S5.T1) and [Table 2](https://arxiv.org/html/2509.15579v1#S5.T2) are initialized with dedicated pre-trained models either offline or streaming, while a Chunk SSL model could be initialized with the same pre-trained model and operated in both modes.

In the speech translation experiments, we evaluate on both “En$\rightarrow$Es” and “En$\rightarrow$De” directions and the results are listed in [Table 3](https://arxiv.org/html/2509.15579v1#S5.T3). The baselines are initialized with a speech recognition acoustic encoder (“ASR” in the column “Init.”), which is trained with the English data in the MuST-C English-Spanish direction. Both “En$\rightarrow$Es” and “En$\rightarrow$De” results show that the encoder initialized with Chunk SSL outperforms the one initialized with a speech recognition encoder trained with Must-C data only. We could draw similar conclusion as the Librispeech experiment that the Chunk SSL could effectively improve speech translation quality for both streaming and offline cases.

**Table 4: Comparison of finite scalar quantization with different vocabulary sizes. “phn pur.” and “PNMI” stands for phone purity and phone-normalized mutual information.**
| vocab size | time(s) | levels | phn pur. | PNMI | dev-clean | dev-other |
| --- | --- | --- | --- | --- | --- | --- |
| 1000* | 1990 | 4 ($8\times 1-5\times 3$) | 0.19 | 0.01 | 2.1 | 5.5 |
| 1000 | 1988 | 4 ($8\times 1-5\times 3$) | 0.19 | 0.01 | 2.2 | 5.4 |
| 50,625 | 1998 | 8 ($5\times 4-3\times 4$) | 0.22 | 0.22 | 2.1 | 5.3 |
| 1,265,625 | 2039 | 10 ($5\times 6-3\times 4$) | 0.67 | 0.83 | 2.0 | 5.2 |
| 6,834,375 | 2240 | 12 ($5\times 5-3\times 7$) | 0.89 | 0.95 | 2.0 | 5.0 |
| 791,015,625 | 2250 | 14 ($5\times 10-3\times 4$) | 0.97 | 0.99 | 2.0 | 5.4 |

### 5.2 Impact of FSQ codebook sizes

In order to verify our a high resolution FSQ codebook hypothesis discussed in [Section 3](https://arxiv.org/html/2509.15579v1#S3),
we study FSQ tokenizers with different codebook sizes. Two aspects are examined: first the agreement among FSQ tokens and phonemes, and second WERs for ASR models initialized with different FSQ tokenizers.
For the agreement among FSQ tokens and phonemes, we compare the overlap of features associated with different tokens and phonemes. It can be measured by phone purity and phone-normalized mutual information (PNMI) (Hsu et al., 2021a). Phone purity denotes the conditional probability of a phoneme given a FSQ index while PNMI measures the percentage of uncertainty about a phoneme label eliminated after observing a specific FSQ token.
We generate phoneme based alignment with Montreal Aligner (McAuliffe et al., 2017)(^3^33https://github.com/MontrealCorpusTools/mfa-models/releases/download/acoustic-english_us_arpa-v3.0.0/english_us_arpa.zip)
on two Librispeech dev sets.

[Table 4](https://arxiv.org/html/2509.15579v1#S5.T4) lists models trained with different codebook sizes with the base configure. The first column gives FSQ vocabulary sizes, the second column provides the Chunk SSL training time (in seconds) for 1000 updates.
“levels” is the FSQ levels for quantization. For example, “4($8\times 1-5\times 3$)” in the first row means there are 4 output channels, the first channel is with level 8 and the remaining 3 channels are with level 5. They span a codebook with size 1000.
“phn pur.” stands for phone purity.

The first two rows listed models trained with FSQ vocabulary size 1000. “1000*” means the model is optimized with a full codebook as [Eq. 4](https://arxiv.org/html/2509.15579v1#S3.E4), while “1000” indicates the model is pretrained with the group masked prediction loss following [Eq. 5](https://arxiv.org/html/2509.15579v1#S3.E5). The results shows two models
achieve comparable accuracies
with similar pre-training time.
As shown in [Table 4](https://arxiv.org/html/2509.15579v1#S5.T4), when the codebook size increases, “phn pur.” and “PNMI” increase steadily.
Both metrics show tokens in the codebook with a high resolution have better consistency with phonemes. This confirms our assumption that frames associated with a specific FSQ token are more likely being labelled with the same phoneme if the codebook resolution is high enough.
On the other hand, we observe WER is reduced when FSQ codebook size becomes bigger.
It is peaked for the codebook size around 6 millions.
When the codebook size gets extremely big, i.e. 791 millions in [Table 4](https://arxiv.org/html/2509.15579v1#S5.T4), the “dev-other” WER increases from 5.0 to 5.4.
Our hypothesis is that an extremely large codebook makes the optimization harder, and more training time might be required to get good representation.
Based on above observations, we conclude that a codebook with high resolution helps Chunk SSL to transfer knowledge to the downstream task. However, an extremely large FSQ codebook might make the pre-training hard and hurt the downstream task.

The next thing we examined is computation cost.
We don’t observe any statistical difference for the FSQ training time when the codebook size changes, since majority of computation time is spent on data input/output, encoder and decoder Transformer/Conformer layer computation. On the other hand, a Chunk SSL training with a large codebook can increase the training time slightly that we find
about 10% training time increase when the codebook size grows from 1000 to 6 millions as shown in [Table 4](https://arxiv.org/html/2509.15579v1#S5.T4).

### 5.3 Latency evaluation

We use the base Transducer model to analyze the impact of different chunk sizes on latency and performance. We use Length-Adaptive Average Lagging(LAAL) (Papi et al., 2022) as latency scorer, which is evaluated with the Simuleval toolkit (Ma et al., 2020). We employ greedy decoding instead of beam search decoding for simplicity in this study.

Figure: (a) Librispeech dev-other WER and Length Adaptive Average Lagging (LAAL) vs chunk size (ms)
Refer to caption: figures/librispeech_wer_vs_latency.png

[Figure 3](https://arxiv.org/html/2509.15579v1#S5.F3) depicts the model performance (blue) and latency (red) under different chunk sizes for speech recognition ([3(a)](https://arxiv.org/html/2509.15579v1#S5.F3.sf1)) and speech translation ([3(b)](https://arxiv.org/html/2509.15579v1#S5.F3.sf2) and [3(c)](https://arxiv.org/html/2509.15579v1#S5.F3.sf3)). The speech recognition is evaluated on the Librispeech dev-other set and speech translation tasks are evaluated on the corresponding MuST-C dev set.
It can be seen that the model is with a small latency (LAAL 0.47 second) but high WER (7.6) when the chunk size is small (160ms). The performance improves steadily when the chunk size increases but at the cost of increasing latency. The speech translation experiments have similar observations. The results indicate the model can switch from streaming mode to offline mode smoothly as the chunk sizes change.
Other latency evaluation results, such as Average Lagging (AL) (Ma et al., 2019) and Differentiable Average Lagging (DAL) (Arivazhagan et al., 2019), could be found in [Appendix B](https://arxiv.org/html/2509.15579v1#A2)

## 6 Related Work

Speech self-supervised learning algorithms form contextual representations through learning to predict
missing information due to time constraint or masking. van den Oord et al. (2018) propose an auto-regressive based SSL approach to predict the future samples based on the previous samples. Speech SSL methods such as wav2vec2 (Baevski et al., 2020b), HuBert (Hsu et al., 2021b) and BEST-RQ (Chiu et al., 2022) choose to randomly mask audio span and the model learns to restore those masked regions. Chunk SSL combines both approaches that it masks audio span in the extended chunk
and recovers masking frame indices in an chunk based auto-regressive manner.

Vector quantization (VQ) is one of the important components in the speech SSL. Baevski et al. (2020a) leverages Gumbel-Softmax with multiple variable groups to discretize
audio features. Methods such as diversity loss have to take to alleviate codeword collapse issue.
Hsu et al. (2021b) resorts k-means to cluster the encoder outputs to build the codebook. In order to achieve a good representation, an iterative training is required to keep re-building the encoder with a new codebook.
BEST-RQ (Chiu et al., 2022) chooses an alternative way, which initializes the codebook randomly and no further update is applied during SSL.
Compared with these VQ methods aforementioned, FSQ does not suffer from codebook collapse and achieves high codebook usage (Mentzer et al., 2024).

## 7 Conclusions

In this work, a chunkwise speech self-supervised learning approach is proposed. The model is trained to
learn to reconstruct the masked information in the right most chunk based on unmasked frames in the same chunk and preceding chunks. An efficient copy and append data augmentation method is proposed to
parallel chunk-wise computation.
FSQ is employed to generate high resolution discrete tokens and a group based decomposition method is proposed to alleviate the computation challenge.
The model is trained with varied chunk durations to fit
the different scenarios, hence a pre-trained model could be used for both streaming and offline
applications. Our experiments show that a model initialized with Chunk SSL could achieve very
competitive offline results and excellent streaming results on Librispeech and two MuST-C translation directions.

## References

- Arivazhagan et al. (2019)
N. Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel.
Monotonic infinite lookback attention for simultaneous machine translation.
In *Annual Meeting of the Association for Computational Linguistics*, 2019.
- Baade et al. (2025)
Alan Baade, Puyuan Peng, and David Harwath.
Syllablelm: Learning coarse semantic units for speech language models.
*ICLR*, 2025.
- Baevski et al. (2020a)
Alexei Baevski, Steffen Schneider, and Michael Auli.
vq-wav2vec: Self-supervised learning of discrete speech representations.
In *ICLR*, 2020a.
- Baevski et al. (2020b)
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.
wav2vec 2.0: A framework for self-supervised learning of speech representations.
In *NeurIPS*, 2020b.
- Chen et al. (2021)
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Micheal Zeng, and Furu Wei.
Wavlm: Large-scale self-supervised pre-training for full stack speech processing.
*IEEE Journal of Selected Topics in Signal Processing*, 16, 2021.
- Chiu et al. (2022)
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu.
Self-supervised learning with random-projection quantizer for speech recognition.
In *International Conference on Machine Learning*, 2022.
- Cho & Esipova (2016)
Kyunghyun Cho and Masha Esipova.
Can neural machine translation do simultaneous translation?
*ArXiv*, 2016.
- Chung et al. (2018)
Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James R. Glass.
Unsupervised cross-modal alignment of speech and text embedding spaces.
In *NeurIPS*, 2018.
- Dong & Xu (2020)
Linhao Dong and Bo Xu.
CIF: Continuous integrate-and-fire for end-to-end speech recognition.
*ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020.
- Dong et al. (2020)
Linhao Dong, Cheng Yi, Jianzong Wang, Shiyu Zhou, Shuang Xu, Xueli Jia, and Bo Xu.
A comparison of label-synchronous and frame-synchronous end-to-end models for speech recognition.
*ArXiv*, abs/2005.10113, 2020.
- Fu et al. (2024)
Biao Fu, Kai Fan, Minpeng Liao, Yidong Chen, Xiaodong Shi, and Zhongqiang Huang.
wav2vec-s: Adapting pre-trained speech models for streaming.
In *Annual Meeting of the Association for Computational Linguistics*, 2024.
- Gangi et al. (2019)
Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi.
MuST-C: a multilingual speech translation corpus.
In *NAACL-HLT*, 2019.
- Graves (2012)
Alex Graves.
Sequence transduction with recurrent neural networks.
*ArXiv*, 2012.
- Graves et al. (2013)
Alex Graves, Abdel rahman Mohamed, and Geoffrey E. Hinton.
Speech recognition with deep recurrent neural networks.
*2013 IEEE International Conference on Acoustics, Speech and Signal Processing*, 2013.
- Gulati et al. (2020)
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.
Conformer: Convolution-augmented transformer for speech recognition.
*Interspeech*, 2020.
- Hsu et al. (2021a)
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.
Hubert: Self-supervised speech representation learning by masked prediction of hidden units.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 2021a.
- Hsu et al. (2021b)
Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed1.
Hubert: How much can a bad teacher benefit asr pre-training.
In *ICASSP*, 2021b.
- Kahn et al. (2019)
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazar’e, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdel rahman Mohamed, and Emmanuel Dupoux.
Libri-light: A benchmark for asr with limited or no supervision.
*ICASSP*, 2019.
- Kano et al. (2023)
Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura.
Average token delay: A latency metric for simultaneous translation.
*Interspeech*, 2023.
- Kim & Rush (2016)
Yoon Kim and Alexander M. Rush.
Sequence-level knowledge distillation.
In *Conference on Empirical Methods in Natural Language Processing*, 2016.
- Kudo & Richardson (2018)
T. Kudo and J. Richardson.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.
In *EMNLP*, 2018.
- Langman et al. (2024)
Ryan Langman, Ante Juki’c, Kunal Dhawan, Nithin Rao Koluguri, and Jason Li.
Spectral codecs: Improving non-autoregressive speech synthesis with spectrogram-based audio codecs.
*ArXiv*, abs/2406.05298, 2024.
- Li et al. (2023)
Xilai Li, Goeric Huybrechts, S. Ronanki, Jeffrey J. Farris, and S. Bodapati.
Dynamic chuck convolution for unified streaming and non-streaming conformer asr.
In *ICASSP*, 2023.
- Liu et al. (2021)
Dan Liu, Mengge Du, Xiaoxi Li, Ya Li, and Enhong Chen.
Cross attention augmented transducer networks for simultaneous translation.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, 2021.
- Ma et al. (2019)
Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang.
Stacl: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework.
In *Annual Meeting of the Association for Computational Linguistics*, 2019.
- Ma et al. (2020)
Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, and Juan Pino.
Simuleval: An evaluation toolkit for simultaneous translation.
*arXiv preprint arXiv:2007.16193*, 2020.
- McAuliffe et al. (2017)
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger.
Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi.
In *Interspeech*, 2017.
- Mentzer et al. (2024)
Fabian Mentzer, David C. Minnen, Eirikur Agustsson, and Michael Tschannen.
Finite scalar quantization: Vq-vae made simple.
*ICLR*, 2024.
- Panayotov et al. (2015)
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur.
Librispeech: An ASR corpus based on public domain audio books.
In *ICASSP*, 2015.
- Papi et al. (2022)
Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi.
Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation.
*ACL*, 2022.
- Park et al. (2019)
D. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. Cubuk, and Q. Le.
Specaugment: A simple data augmentation method for automatic speech recognition.
*Interspeech*, 2019.
- Shaw et al. (2018)
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.
Self-attention with relative position representations.
In *North American Chapter of the Association for Computational Linguistics*, 2018.
- Shi et al. (2021)
Yangyang Shi, Yongqiang Wang, C. Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, D. Le, and Mike Seltzer.
Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition.
In *ICASSP*, 2021.
- Tang et al. (2022)
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdel rahman Mohamed, Michael Auli, and Juan Miguel Pino.
Unified speech-text pre-training for speech translation and recognition.
In *ACL*, 2022.
- Tang et al. (2023)
Yun Tang, Anna Sun, Hirofumi Inaguma, Xinyue Chen, Ning Dong, Xutai Ma, Paden Tomasello, and Juan Miguel Pino.
Hybrid transducer and attention based encoder-decoder modeling for speech-to-text tasks.
In *Annual Meeting of the Association for Computational Linguistics*, 2023.
- Tsunoo et al. (2019)
Emiru Tsunoo, Yosuke Kashiwagi, Toshiyuki Kumakura, and Shinji Watanabe.
Transformer asr with contextual block processing.
*2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)*, pp. 427–433, 2019.
- van den Oord et al. (2018)
Aäron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
*ArXiv*, 2018.
- Vaswani et al. (2017)
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In *Neural Information Processing Systems*, 2017.
- Weninger et al. (2022)
Felix Weninger, Marco Gaudesi, Md. Akmal Haidar, Nicola Ferri, Jes’us Andr’es-Ferrer, and Puming Zhan.
Conformer with dual-mode chunked attention for joint online and offline asr.
In *Interspeech*, 2022.
- Zhang et al. (2020a)
Binbin Zhang, Di Wu, Zhuoyuan Yao, Xiong Wang, F. Yu, Chao Yang, Liyong Guo, Yaguang Hu, Lei Xie, and Xin Lei.
Unified streaming and non-streaming two-pass end-to-end model for speech recognition.
*ArXiv*, abs/2012.05481, 2020a.
- Zhang et al. (2020b)
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar.
Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss.
In *ICASSP*, 2020b.
- Zhang et al. (2020c)
Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, and Yonghui Wu.
Pushing the limits of semi-supervised learning for automatic speech recognition.
*ArXiv*, abs/2010.10504, 2020c.

## Appendix A Optimization configures

### A.1 FSQ pre-training

The FSQ model is built with learning rate is 2.0e-4 and ExponentialLR scheduler from PyTorch. The effective batch size is 3200 seconds with up to 180,000 updates. The FSQ module is optimized with MSE loss to restore mel-filterbank features from their quantized audio representations. It takes 4 A10 GPUs for 42 hours to build a FSQ model.

### A.2 Chunk SSL pre-training setting

The pre-training is scheduled as Transformer (Vaswani et al., 2017) with warmup step 25,000 and maximum learning rate 3e-4. The batch size is 16,000 seconds for the base configure model. We choose a smaller effective batch size for the large configure model with 6,000 seconds due to computation resource limitation. The model is trained up to 400,000 updates. It takes about 10 days for 8 A100 GPUs to build a Chunk SSL encoder with the base configure.

### A.3 Librispeech fine-tuning setting

960 hours fine-tuning
For the base configure model, we choose the three stages scheduler scheme as (Baevski et al., 2020b), i.e., warmup, hold and annealing. The warmup stage takes about 1000 updates with encoder parameters frozen, the hold stage takes about 40,000 updates and the remaining 60,000 updates are in an annealing stage. The peak learning rate in fine-tuning is 2.0e-3.
The SpecAugment (Park et al., 2019) data augmentation is applied with 2 frequency maskings of 27, and 10 time maskings at a maximum ratio of 0.05 in fine-tuning experiments.
The effective batch size for fine-tuning is 10,000 seconds. For the large configure model, we choose Transformer scheduler (Vaswani et al., 2017) since it has less hype-parameters to explore. The warmup step is 5000 with encoder parameters frozen. Similar as the base configure model, the total update step is up to 100,000. The peak learning rate is 5e-4. The effective batch size for the large model is 7200 seconds.

### A.4 MuST-C fine-tuning settings

The translation Transducer model is optimized with Transformer (Vaswani et al., 2017) scheduler with warmup step 10,000 and total update step up to 100,000. The peak learning rate is 5e-4. The effective batch size is 6400 seconds.

## Appendix B Latency Results

**Table 5: Latency and WER vs chunk size on base model trained with Librispeech evaluated on dev-other.**
| Chunk Size (ms) | WER | LAAL | AL | AP | DAL | ATD |
| --- | --- | --- | --- | --- | --- | --- |
| 160 | 7.6 | 469.1 | -1753.6 | 1.0 | 900.1 | 760.6 |
| 320 | 6.8 | 682.6 | -1492.2 | 1.1 | 1133.9 | 908.5 |
| 640 | 6.0 | 1161.3 | -833.6 | 1.2 | 1686.8 | 724.9 |
| 960 | 5.7 | 1627.8 | -177.9 | 1.3 | 2223.4 | 785.5 |
| 1280 | 5.5 | 2106.2 | 492.3 | 1.4 | 2754.8 | 928.1 |
| 1600 | 5.4 | 2566.1 | 1116.8 | 1.5 | 3246.2 | 1083.3 |
| 1920 | 5.4 | 2988.4 | 1686.7 | 1.5 | 3678.4 | 1230.6 |
| 2240 | 5.3 | 3373.7 | 2215.2 | 1.6 | 4058.9 | 1365.1 |

**Table 6: Latency and BLEU vs chunk size on base model trained with MustC EN-DE dev dataset.**
| Chunk Size (ms) | BLEU | LAAL | AL | AP | DAL | ATD |
| --- | --- | --- | --- | --- | --- | --- |
| 320 | 18.3 | 782.0 | -2475.3 | 1.4 | 1329.9 | 447.8 |
| 640 | 19.7 | 1208.5 | -1841.5 | 1.6 | 1826.4 | 440.0 |
| 960 | 20.3 | 1635.9 | -1188.3 | 1.7 | 2308.0 | 538.9 |
| 1280 | 20.7 | 2056.2 | -534.5 | 1.8 | 2779.6 | 665.1 |
| 1600 | 20.9 | 2455.6 | 91.6 | 1.8 | 3190.3 | 781.8 |
| 1920 | 21.0 | 2803.6 | 625.1 | 1.9 | 3558.3 | 904.7 |
| 2240 | 21.1 | 3137.0 | 1141.6 | 1.9 | 3882.8 | 1010.6 |

**Table 7: Latency and BLEU vs chunk size on base model trained with MustC EN-ES dev dataset.**
| Chunk Size (ms) | BLEU | LAAL | AL | AP | DAL | ATD |
| --- | --- | --- | --- | --- | --- | --- |
| 320 | 24.4 | 680.4 | -2601.2 | 1.3 | 1277.3 | 495.7 |
| 640 | 26.1 | 1115.2 | -1905.6 | 1.5 | 1791.4 | 461.7 |
| 960 | 27.0 | 1561.8 | -1248.4 | 1.6 | 2297.2 | 559.9 |
| 1280 | 27.0 | 1979.6 | -607.8 | 1.7 | 2782.1 | 680.3 |
| 1600 | 27.2 | 2402.3 | 33.2 | 1.8 | 3228.4 | 819.5 |
| 1920 | 27.2 | 2781.1 | 584.0 | 1.8 | 3630.6 | 940.3 |
| 2240 | 27.3 | 3131.4 | 1106.7 | 1.9 | 3990.4 | 1047.7 |

The full latency evaluation results including Average Lagging (AL) (Ma et al., 2019), Length Adaptive Average Lagging (LAAL) (Papi et al., 2022),
Average Proportion (AP) (Cho & Esipova, 2016), Differentiable Average Lagging (DAL) (Arivazhagan et al., 2019), and Average Token Delay (ATD) (Kano et al., 2023) for the Librispeech and Must-C are presented in [Table 5](https://arxiv.org/html/2509.15579v1#A2.T5), [Table 6](https://arxiv.org/html/2509.15579v1#A2.T6) and [Table 7](https://arxiv.org/html/2509.15579v1#A2.T7).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: **[Q1] Questions regarding the presentation:**

[Q1-1] In L74-76, *"We hypothesize that speech frames associated with a high resolution FSQ token are mainly mapped to an unique modeling unit in the downstream task and it makes the knowledge transfer easier from the pre-training stage to the fine-tuning stage."* 
What does this sentence mean? What is *"an unique modeling unit in the downstream task"*? Do you have any support on this?

[Q1-2] In L106-109, *"Inspired by the implementation of the chunk encoder with a look-ahead chunk for the streaming speech translation (Liu et al., 2021), we propose a copy and append data augmentation for the Chunk SSL, which reorganizes input data and changes the augmented data computation, but it is still strictly equivalent to the computation in the naive Chunk SSL algorithm."* 
I am unsure which part of Liu et al.’s work inspired you, and what exactly the ‘naive chunk SSL algorithm’ refers to (seems to be a critical baseline).

[Q1-3] In L115-118, *"Assuming the chunk size is 4 frames, the speech features are segmented into three base chunks (in cyan): “B1”,
“B2” and “B3”. Chunk “B2” and “B3” are copied and appended to the end of the utterance as “E1” and “E2” (in lime). They correspond to the extended chunks of “B1” and “B2” respectively. There is no extended chunk for the last chunk (“B3”)."* 
The statement that chunks “B2” and “B3” are copied and appended (extended) contradicts the claim that there is no extended chunk for the last chunk “B3.”

[Q1-4] Overall, I cannot quite get why we need to ‘copy and append,’ and how intuitively effective this approach can be, given that the *naive chunk SSL algorithm* is not well introduced or clearly defined as a baseline.

[Q1-5] In L135, what is X? You should define the transformation from X to X′ (the CADA-augmented utterance) more precisely and carefully. 

[Q1-6] Figure 2-(b) is confusing. In the main text that discuss it, why does "B1" pair with "E1" (L203)? Shouldn’t "B2" pair with "E1," according to Figure 1?

---

**[Q2] Questions about the soundness of experiments**

[Q2-1] In Table 1 and Table 2, only the Chunk SSL scores are reported. However, to demonstrate the effectiveness of the proposed method (CADA), the so-called *"naive Chunk SSL"* should be included as a baseline. 

[Q2-2] Following the previous question, how much benefit is gained from using FSQ?

[Q2-3] Why does Table 3 not include other SSL methods?

[Q2-4] Are all baselines in Tables 1 and 2 taken from the literature? If so, how do you ensure the consistency of experiments between the previous work and your work? Has BEST-RQ released its codebase?

---

**[Q3] Questions regarding the core design of CADA**

[Q3-1] Can you elaborate on the novelty of CADA? What do you think is beyond "technical training efficiency improvements"?

[Q3-2] What is "copy and append" mainly about? Is it about 1. enabling look ahead and 2. getting the target from the base output for the extended chunks?

[Q3-3] If it is all about the two aforementioned points, why not simply apply batch expansion? We can have the first half of the batch unmasked and have the look-ahead causal chunked attention masks, while the second half (also copied from the first half) partially masked and having the corresponding causal chunked mask that is equivalent to CADA's design. The batch-expanded version seems more natural to me, since it requires no length expansion for each utterance and avoids specialized convolution operations (split and concat in Figure 2-(b)).
soundness: 2
strengths: [S1]: The motivation is clear. The chunkwise SSL could intuitively help on the streaming performance of the specified downstream tasks. 

[S2]: The method appears to be training-efficient.
summary: This work proposes a chunkwise pretraining scheme for speech self-supervised learning (SSL), aiming to enhance streaming capability for downstream tasks, including automatic speech recognition (ASR) and speech-to-text translation (ST). The authors develop a training scheme that allows efficient training of the chunkwise speech SSL by using the specialized "copy and append data augmentation (CADA)" technique. In addition, they employ a masked-prediction objective using finite scalar quantization (FSQ) codes as targets. They evaluate their Chunk SSL speech models on ASR and ST, reporting both offline (non-streaming) and online (streaming) performance.
weaknesses: [W1]: The presentation requires further refinement. I cannot fully grasp the precise meaning of some sentences. Even the context of the core design—CADA—is somewhat confusing. See [Q1].

[W2]: The completeness and soundness of the experiments are questionable. The authors propose CADA and FSQ on speech SSL, but I cannot infer how effective CADA or FSQ are individually from the provided results. See [Q2].

[W3]: The novelty is limited. The novelty mainly lies in CADA. However, I consider CADA as a technique for efficiently train the chunkwise SSL model, at the cost of increased memory usage due to the "copy-and-append" behavior. The idea of trading memory for time is not particularly novel or impressive. I also have a few questions regarding the core design choices; please see [Q3].

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. What is the motivation for introducing an additional FSQ module for target prediction instead of directly using mel-filterbank features?

2. In the non-streaming training setup, how is masked prediction performed if the first chunk is not appended?

3. Have you explored training a unified model for both streaming and non-streaming scenarios without the copy and append operation?
soundness: 3
strengths: 1. The paper introduces a model capable of dynamically adjusting the chunk size during both training and inference, enabling users to flexibly control latency and reducing the mismatch between training and inference.

2. The proposed CADA approach might be a potential alternative to previous masking strategies for self-supervised prediction, e.g. Best-RQ and Wav2vec 2.0.
summary: The paper proposes an integrated framework for both non-streaming and streaming self-supervised speech training. By introducing a new CADA masking strategy and FSQ-based target prediction, the unified model achieves performance comparable to prior specialized models for either setting. However, additional ablation studies are required to justify the benefit and usage of CADA and FSQ. In particular, CADA might be replicable by simply applying both non-causal and block-wise causal attention masks during self-supervised training, without the need for copy and append. Likewise, a comparison for adopting FSQ over continuous targets would greatly justify its use.
weaknesses: 1. If the goal is to obtain fine-grained prediction targets, why not consider continuous representation could serve as a more efficient alternative, e.g. mel-filterbank features. The benefit of quantizing mel-filterbank features using FSQ for mask prediction is not clearly established. Since FSQ quantizes each channel independently, it is reminiscent of the approach in dMel: Speech Tokenization Made Simple. However, it seems plausible that continuous mel-filterbank features could suffice for training. A direct comparison between models predicting continuous mel-filterbank features and those using FSQ targets would be necessary to justify the added complexity of introducing an FSQ encoder.

2. As shown in Figure 2(b), it is unclear why the “1st chunk” is not appended as well. Assigning zero attention scores to the 'full context 1st chunk' to the 'appended 1st chunk' should work.

3. Following above, when using “offline training with infinite chunk size", Figure 2 shows no indication of the masked chunk being copied and appended for prediction, which does not seem right.

4. A more thorough comparison with simple joint causal and non-causal masking during training would strengthen the justification for using CADA. For example, training a Best-RQ-style alternative that includes both chunk-wise causal attention mask and non-causal attention masks within each batch would be a fair baseline. This could also justify CADA using twice the computation during training.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - I see clear gains for streaming ASR, but the offline WERs trail the comparative approach. Could you explain why? Intuitively, when the streaming constraints are relaxed, an effective method should at least match—ideally surpass—offline baselines.
- As a follow-up: if CADA is strictly equivalent to naïve chunk SSL (only enabling parallelization), does that imply the conventional naïve approach could also support streaming—and, if trained identically, would its offline performance remain competitive?
- Please also address the weaknesses noted above. I’m happy to raise my score if these concerns are satisfactorily resolved.
soundness: 3
strengths: - The paper proposes an alternative streaming SSL recipe that aims to unify pretraining for both streaming and offline speech encoders within a single framework
summary: The paper proposes Chunk SSL, a chunk-wise self-supervised pretraining framework that aims to unify streaming and offline speech encoders. Core pieces: (1) Copy-and-Append Data Augmentation (CADA) to parallelize chunked masked-prediction pretraining; (2) Finite Scalar Quantization (FSQ) to discretize frames, with very large codebooks; and (3) a group masked-prediction loss to keep compute/memory manageable with high-resolution codebooks.
weaknesses: - The method largely recombines familiar pieces—chunked SSL, discretization, and masking—with the main novelty in CADA and the FSQ + group-loss design.
- Given how widely SSL embeddings are used, the paper’s downstream evaluation feels narrow; broader tasks (e.g. speaker verification/diarization) would better demonstrate generality.
- Although effective, the approach depends on millions-level FSQ codebooks; as the vocabulary grows, optimization becomes harder and WER degrades—raising concerns about training stability and compute cost.
- Some implementation details for CADA are hard to follow; it would help to include a concise algorithm box/pseudocode.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: My primary questions are already listed in the Weaknesses field. If these questions are addressed satisfactorily, I am open to revising my scores.

I also have a few minor questions and suggestions that could improve the clarity and comparability of the paper:

- Why was Conformer chosen as the backbone? Many SSL models (e.g., Wav2Vec 2.0, HuBERT, WavLM) are based on standard Transformers, which would allow for a clearer comparison.
- Similarly, regarding model size, it would improve comparability if the proposed method used a similar number of parameters as the baseline models.
soundness: 3
strengths: - The paper presents an efficient implementation of chunk-based SSL, which is particularly valuable for streaming ASR and ST applications where low-latency processing is critical.
- The paper also provides an efficient discretization by using multi-label classification.
- The proposed approach demonstrates strong experimental performance, outperforming existing SSL-based methods on both ASR and ST benchmarks.
summary: This paper proposes a chunk-based self-supervised learning (SSL) framework that masks both future chunk information and sampled frames within each chunk. The method is efficiently implemented through copy-and-append data augmentation techniques, enabling scalable training. In addition, the paper introduces a multi-label prediction scheme, inspired by audio codecs, to efficiently handle large vocabulary sizes. Training is conducted using dynamic chunk sizes, allowing the model to learn representations across varying temporal contexts, while inference leverages the chunk structure to balance latency and performance. Experimental results on LibriSpeech ASR and MuST-C speech translation tasks demonstrate the effectiveness of the proposed approach.
weaknesses: * **Unclear masking implementation (potentially critical):**
  The manuscript’s description of the masking operation is ambiguous. The text appears to apply masks by multiplying masked positions by **0** rather than using large negative values before the softmax (or otherwise excluding them from the attention denominator). If true, this would leave nonzero attention weights on masked/future positions (the softmax denominator still includes those entries), allowing the model to attend to information that should be hidden. This is a fundamental concern that could invalidate the reported results. Please clarify precisely how masking is implemented (include equations), and provide an ablation that compares the current implementation with a standard masked-attention formulation (e.g., add −∞ or a large negative constant before softmax) to show the effect on performance and verify correctness.

* **Limited novelty and missing literature context for multi-label representations:**
  The multi-label / codec-like representation component echoes several recent works on discrete and multi-label audio representations (for example, MMM and other codec-inspired SSL approaches). The paper currently does not situate its contribution clearly within this literature nor convincingly establish novelty. Please expand the related-work discussion to cover recent multi-label and codec-based representation studies, and clarify what is new here (algorithmic change, efficiency, empirical regime, etc.). If novelty is incremental, the manuscript should present stronger empirical or analytical evidence that the proposed variant offers a meaningful advantage over prior methods.
  * Shi, Jiatong, et al. "MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model." Proc. Interspeech 2024. 2024.

* **Insufficient analysis of representation trade-offs (phoneme purity alone is inadequate):**
  The analysis relies heavily on phoneme-label purity as a measure of representational quality. However, phoneme purity alone favors larger codebook / vocabulary sizes and does not capture clustering compactness or redundancy trade-offs. The paper should include additional probing analyses based on **cluster purity**, and report trade-offs between vocabulary size, cluster purity, and downstream performance. Presenting such metrics (and plotting cluster/phoneme purity for each vocabulary size) would provide a more trustworthy and nuanced interpretation of why larger vocabularies help or hurt.


Decision: Reject