CR-CTC: Consistency regularization on CTC for improved speech recognition

## Abstract

Abstract Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR), renowned for its simplicity and computational efficiency. However, it often falls short in recognition performance compared to transducer or systems combining CTC and attention-based encoder-decoder (CTC/AED).
In this work, we propose the Consistency-Regularized CTC (CR-CTC), which enforces consistency between two CTC distributions obtained from different augmented views of the input speech mel-spectrogram.
We provide in-depth insights into its essential behaviors from three perspectives: 1) it conducts self-distillation between random pairs of sub-models that process different augmented views; 2) it learns contextual representation through masked prediction for positions within time-masked regions, especially when we increase the amount of time masking; 3) it suppresses the extremely peaky CTC distributions, thereby reducing overfitting and improving the generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of our CR-CTC, which achieves performance comparable to, or even slightly better than, that of transducer and CTC/AED.
We release our code at https://github.com/k2-fsa/icefall .

## 1 introduction

End-to-end approaches (Graves et al., ; Graves, ; Chan et al., ), which eliminate the need of pre-aligned speech-text data, have replaced traditional hybrid systems (Bourlard & Morgan, ; Hinton et al., ) and become dominant methods in automatic speech recognition (ASR).
Prominent examples include Connectionist Temporal Classification (CTC) (Graves et al., ), Transducer (Graves, ) (also known as RNN-T), and the method that combines CTC and attention-based encoder-decoder (AED) (Chan et al., ), referred to as CTC/AED (Watanabe et al., ).
To handle the alignment between speech and token sequences, CTC (Graves et al., ) introduces a blank token and makes independent predictions at each frame, training the model to maximize the total probability over all valid alignments.
Transducer (Graves, ) extends CTC by introducing a prediction network and a joint network, explicitly modeling the interdependencies on output labels.
CTC/AED (Watanabe et al., ) integrates CTC into AED (Chan et al., ) for jointly training, while the CTC and AED scores are fused during the decoding process.
Among these three methods, CTC is the simplest and most computationally efficient due to its frame-independent assumption, making it a strong candidate for real-world deployment. However, it significantly lags behind transducer and CTC/AED in terms of recognition performance, which limits its applicability.

To improve the CTC performance, in this work we propose the Consistency-Regularized CTC (*CR-CTC*), which takes two different augmented views of the same speech mel-spectrogram as input, and enforces consistency between the resulting CTC distributions.
We analyze its internal behaviors from three following perspectives.
First, it performs self-distillation between sub-models randomly sampled by drop-based techniques (Srivastava et al., ; Huang et al., ).
Second, for positions within time-masked regions, the model is required to predict the target token distributions, forcing it to learn contextual representation based on unmasked context, akin to self-supervised learning methods (Devlin et al., ; Baevski et al., ; Hsu et al., ). Therefore, we especially increase the amount of time masking in *CR-CTC* to enhance this masked prediction behavior.
Third, the consistency regularization suppresses extremely peaky CTC distributions, which mitigates overfitting and improves the model’s generalization ability. Inspired by this, we additionally propose an simple method specifically designed to learn smoother CTC distributions (Appendix Section A.1), which is experimentally validated to be effective.

We conduct experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets using Zipformer (Yao et al., ) as speech encoder. The results demonstrate the superiority of *CR-CTC*, which significantly outperforms vanilla CTC and achieves results comparable to, or even slightly better
than, those of transducer and CTC/AED. In addition, *CR-CTC* can further improve the performance of transducer and CTC/AED when employed for jointly training. We perform detailed ablation studies on LibriSpeech dataset to investigate the effect of each functional component in *CR-CTC* and to validate our explanations.

## 2 related work

Self-distillation.
Unlike traditional knowledge distillation (Buciluǎ et al., ; Hinton et al., ), which transfers knowledge from a larger and high-capacity teacher model to a smaller student model, self-distillation (Furlanello et al., ; Zhu et al., ; Mobahi et al., ; Allen-Zhu & Li, ) involves learning from a same-architecture model that processes the same training data. This approach enables the model to extract more refined representations and achieve improved performance.
For example, BANs (Furlanello et al., ) introduces a re-training procedure in which a newly initialized student model is trained to match a pre-trained teacher model, subsequently serving as the teacher in the next iteration.
Some works also explore constructing the teacher and student models from a shared network, distilling knowledge from deeper layers to shallower layers (Zhang et al., ; Kim et al., ), or between pairs of sub-models randomly initialized through drop-based techniques (Srivastava et al., ; Huang et al., ), such as R-Drop (Wu et al., ) and cosub (Touvron et al., ). Our *CR-CTC* fundamentally conducts self-distillation between random sub-models, sharing similar idea to R-Drop and cosub, while our approach further use different augmented input views, which enriches the diversity of predictions from these sub-models.

Masked prediction. Masked prediction has proven highly effective in self-supervised learning (Devlin et al., ; Baevski et al., ; Joshi et al., ; Baevski et al., ; Hsu et al., ; He et al., ; Baevski et al., ). In this approach, the model is tasked with predicting masked positions based on the surrounding unmasked context, which encourages the learning of robust contextual representations. Notable methods for speech representation learning include wav2vec 2.0 (Baevski et al., ), HuBERT (Hsu et al., ), and data2vec 2.0 (Baevski et al., ), which primarily differ in their prediction targets. Specifically, wav2vec 2.0 (Baevski et al., ) jointly trains a representation quantizer and learns to distinguish the true quantized target from distractors (Oord et al., ). HuBERT generates target labels through offline clustering, while data2vec 2.0 uses contextualized representations from a teacher model as its targets.
Our *CR-CTC* essentially performs masked prediction for positions within time-masked regions, where the target labels are frame-level token distributions generated based on another augmented view of input.

Peaky CTC distributions. CTC models are known for predicting extremely peaky distributions (Graves et al., ; Sak et al., ), which can be harmful in certain scenarios, such as forced alignment (Huang et al., ) and knowledge distillation (Ding et al., ).
These peaky distributions lead to inaccurate alignments as the model assigns excessive blanks to non-silence frames.
To address this, label priors are employed to suppress the peaky distributions, thereby improving the accuracy of forced alignment (Huang et al., ).
As position mismatches of CTC spikes can hinder knowledge distillation performance, some approaches propose to encourage consistent alignments between the teacher and student (Ding et al., ) or to utilize sequence-level distillation (Takashima et al., ).
Unlike previous work, we demonstrate that peak suppression in *CR-CTC* can improve the generalization ability of the CTC model.

Consistency regularization. The technique of consistency regularization has demonstrated effectiveness in learning generalizable representations across various learning paradigms, including self-supervised (Chen et al., ; Grill et al., ; Chen & He, ), semi-supervised (Sajjadi et al., ; Laine & Aila, ; Sohn et al., ; Masumura et al., ; Chen et al., ; Higuchi et al., ; Qi et al., ), and supervised (Wu et al., ; Gao et al., ; Touvron et al., ; Heo et al., ; Yoon et al., ) learning tasks.
SimSiam (Chen & He, ), for instance, maximizes the similarity between two augmented views of the same image. FixMatch (Sohn et al., ) generates pseudo-labels based on high-confidence predictions from weakly augmented images, then trains the model to predict these pseudo-labels using the strongly augmented versions of the same images.
Several methods encourage consistency between the predictions of randomly sampled sub-models using drop-based techniques, such as R-Drop (Wu et al., ) and cosub (Touvron et al., ).
The idea of R-Drop has also been applied to CTC models (Graves et al., ) for ASR (Gao et al., ; Yoon et al., ).
For example, in (Gao et al., ), spatial-temporal dropout is specially designed to construct the sub-models, with consistency regularization enforced exclusively on the CTC spike frames.
Similar to our *CR-CTC*, these methods utilizing consistency regularization are also interpreted as performing self-distillation between the random sub-models (Wu et al., ; Touvron et al., ).

## 3 method

We first introduce the standard CTC algorithm in Section 3.1. Then we present the detailed implementation of our proposed Consistency-Regularized CTC (*CR-CTC*) in Section 3.2, followed by in-depth explanations from different perspectives in Section 3.3.

### 3.1 Preliminary: Connectionist Temporal Classification

The ASR task is to convert a sequence of speech frames $\mathbf{x}=\{x_{t}\}_{1}^{T}$ of length $T$ to a sequence of transcript tokens $\mathbf{y}=\{y_{u}\in\mathcal{V}\}_{1}^{U}$ of length $U$, where $\mathcal{V}$ is the vocabulary and typically $T\geq U$. CTC (Graves et al., ) extends the vocabulary $\mathcal{V}$ to $\mathcal{V}^{\prime}=\mathcal{V}\cup\{\epsilon\}$ with a blank token $\epsilon$, and aims to maximize the total posterior probability of all valid alignments $\bm{\pi}=\{\pi_{t}\in\mathcal{V}^{\prime}\}^{T}_{1}$ between $\mathbf{x}$ and $\mathbf{y}$. Let $\mathcal{B}(\bm{\pi})$ denote the many-to-one map that merges repeating tokens and removes all blanks in $\bm{\pi}$, and $p(\bm{\pi}|\mathbf{x})$ denote the posterior probability of alignment $\bm{\pi}$, the CTC loss function is formulated as:

$$ $\mathcal{L}_{\mathrm{CTC}}(\mathbf{x},\mathbf{y})=-\log\sum_{\bm{\pi}\in \mathcal{B}^{-1}(\mathbf{y})}p(\bm{\pi}|\mathbf{x}).$ (1) $$

Specifically, given the input $\mathbf{x}$, it employs an encoder $f$ to estimate the $|\mathcal{V}^{\prime}|$-dimensional probability distributions $\mathbf{z}=\{z_{t}\}^{T}_{1}$: $\mathbf{z}=f(\mathbf{x})$ (^1^11$T$ is typically downsampled in the encoder $f$ by a factor of 4 for efficiency. This is omitted for the sake of simplicity in expression.), where $f$ is modeled by a speech encoder network such as Zipformer (Yao et al., ) followed by a linear projection layer and a *softmax* function.
Under the frame-independent assumption (Graves et al., ), $p(\bm{\pi}|\mathbf{x})$ is computed as:

$$ $p(\bm{\pi}|\mathbf{x})=\prod_{t=1}^{T}z_{t,\pi_{t}},$ (2) $$

where $z_{t,\pi_{t}}$ is the probability of emitting token $\pi_{t}$ at frame $t$.

### 3.2 Our approach: Consistency-regularized CTC

Figure: Figure 1: Overall architecture of *CR-CTC*.
Refer to caption: x1.png

Figure 1 illustrates the overall architecture of our proposed *CR-CTC*. It takes as input two different augmented views, $\mathbf{x}^{(a)}$ and $\mathbf{x}^{(b)}$, both derived from the input speech mel-spectrogram $\mathbf{x}$. The two input views are then passed through a shared speech encoder $f$, which estimates the per-frame distributions: $\mathbf{z}^{(a)}=f(\mathbf{x}^{(a)})$, $\mathbf{z}^{(b)}=f(\mathbf{x}^{(b)})$.
In addition to computing the CTC losses on both branches: $\mathcal{L}_{\mathrm{CTC}}(\mathbf{x}^{(a)},\mathbf{y})$ and $\mathcal{L}_{\mathrm{CTC}}(\mathbf{x}^{(b)},\mathbf{y})$, we introduce an auxiliary loss (defined in Equation 4) to enforce consistency between $\mathbf{z}^{(a)}$ and $\mathbf{z}^{(b)}$: $\mathcal{L}_{\mathrm{CR}}(\mathbf{z}^{(a)},\mathbf{z}^{(b)})$.
The overall loss of the whole model is formulated as:

$$ $\mathcal{L}=\frac{1}{2}(\mathcal{L}_{\mathrm{CTC}}(\mathbf{x}^{(a)},\mathbf{y} )+\mathcal{L}_{\mathrm{CTC}}(\mathbf{x}^{(b)},\mathbf{y}))+\alpha\mathcal{L}_{ \mathrm{CR}}(\mathbf{z}^{(a)},\mathbf{z}^{(b)}),$ (3) $$

where $\alpha$ is a hyper-parameter that controls the consistency regularization.

Different augmented views. The two different augmented views, $\mathbf{x}^{(a)}$ and $\mathbf{x}^{(b)}$, are generated by independently applying SpecAugment (Park et al., ) to two copies of the input mel-spectrogram $\mathbf{x}$. SpecAugment involves warping along time axis, masking blocks of frequency channels, and masking blocks of time steps. Since time warping alters feature timing and thus shifts output timestamps, we apply it first before creating the copies to prevent significant timestamp mismatches between the outputs of two branches. Subsequently, random frequency masking and time masking are both applied to the two copies, resulting in $\mathbf{x}^{(a)}$ and $\mathbf{x}^{(b)}$.
Note that we also increase the amount of time masking by a factor of 2.5 compared to regular systems.
The reason behind this adjustment is explained in Section 3.3, with implementation details provided in Section 4.1.

Consistency regularization loss. The consistency regularization is applied on each frame $t$, by minimizing the bidirectional Kullback-Leibler divergence (denoted as $D_{\mathrm{KL}}$) between each pair of distributions $z^{(a)}_{t}$ and $z^{(b)}_{t}$: $D_{\mathrm{KL}}(\mathrm{\emph{sg}}(z^{(b)}_{t})\|z^{(a)}_{t})$ and $D_{\mathrm{KL}}(\mathrm{\emph{sg}}(z^{(a)}_{t})\|z^{(b)}_{t})$, where sg denotes the operation stopping gradient on the target distributions.
The regularization loss $\mathcal{L}_{\mathrm{CR}}(\mathbf{z}^{(a)},\mathbf{z}^{(b)})$ is formulated as:

$$ $\mathcal{L}_{\mathrm{CR}}(\mathbf{z}^{(a)},\mathbf{z}^{(b)})=\frac{1}{2}\sum_{ t=1}^{T}D_{\mathrm{KL}}(\mathrm{\emph{sg}}(z^{(b)}_{t})\|z^{(a)}_{t})+D_{ \mathrm{KL}}(\mathrm{\emph{sg}}(z^{(a)}_{t})\|z^{(b)}_{t}).$ (4) $$

### 3.3 Explanation

We now explain the essential behaviors of our proposed *CR-CTC* from three different perspectives: 1) it performs self-distillation between pairs of sub-models with different input views; 2) it conducts contextual representation learning by predicting the token distributions at masked positions based on
unmasked context; 3) it suppresses extremely peaky CTC distributions, mitigating overfitting and enhancing generalization ability. We conduct an empirical investigation through ablation studies in Section 4.3, and the experimental results validate our explanations.

Self-distillation.
When using model regularization techniques such as dropout (Srivastava et al., ) and stochastic depth (Huang et al., ), which randomly drop parts of the model (neurons or layers), it can be viewed as implicitly training randomly sampled sub-models that are ultimately combined into an ensemble during inference. Similar to R-Drop (Wu et al., ) and cosub (Touvron et al., ), in *CR-CTC*, enforcing consistency regularization between the two branches enables to perform self-distillation between pairs of randomly sampled sub-models derived from the shared model $f$, with each sub-model receiving supervision signals in the form of per-frame predictions from the other.
In addition, feeding different augmented views (with larger amount of time masking) exposes these sub-models to varied aspects of the input data, enhancing their prediction diversity and facilitating richer knowledge transfer as well as complementary representation learning.

Masked prediction. In *CR-CTC*,
consistency regularization requires frames within the time-masked regions in each branch to predict the corresponding token distributions, which are generated by the other branch on the fly.
Similar to masked-based self-supervised models (Devlin et al., ; Baevski et al., ; Hsu et al., ), this behavior encourages the model to capture acoustic information on the unmasked context and exploit its implicit language modeling capability.
Independently applying random time masking to the two branches reduces the occurrence of positions masked by both branches, thereby improve the quality of the provided target distributions for these masked positions.
Furthermore, increasing the amount of time masking in *CR-CTC* enhances contextual representation learning through the masked prediction behavior.

Peak suppression. In line with previous works (Graves et al., ; Sak et al., ), we also observe that CTC tends to learn extremely peaky distributions. As shown in Figure 2 (left), almost all non-blank tokens occupy only one frame, while the remaining frames are dominated by the blank token, with both types of emissions occurring with extremely high probabilities.
This phenomenon suggests potential overfitting to training data, which limits generalization abality to unseen data.

Enforcing prediction consistency between the two branches in *CR-CTC* guides the model to learn the average of their predictions, ultimately resulting in smoother distributions.
The peak suppression behavior reduces overconfidence on training data, thereby improving the model’s generalization ability.
As presented in Figure 2 (right), *CR-CTC* exhibits reduced token emitting probabilities and an increased occurrence of repeating non-blank tokens. A comparison of concrete statistics on the distribution peakedness between CTC and *CR-CTC* is provided in Table 6.

Inspired by this, we also propose a simple method, called Smooth-Regularized CTC (*SR-CTC*), which incorporates an auxiliary loss into regular CTC, specifically encouraging the model to learn smoother CTC distributions. Appendix Section A.1 presents the details of *SR-CTC*.

Figure: (a) Sample 1 in CTC
Refer to caption: x2.png

## 4 experiments

### 4.1 Experimental setup

Datasets. To evaluate the effectiveness of our proposed *CR-CTC*, we conduct experiments on three publicly available ASR datasets: 1) LibriSpeech (Panayotov et al., ), which contains 1000 hours of English speech; 2) Aishell-1 (Bu et al., ), which consists of 170 hours of Mandarin speech; 3) GigaSpeech (Chen et al., ), comprising 10000 hours of English speech.

Implementation details.
Our experiments are performed using the icefall framework (^2^22[https://github.com/k2-fsa/icefall](https://github.com/k2-fsa/icefall)), with
Lhotse toolkit (Żelasko et al., ) for data preparation. For regular ASR recipes in icefall, default parameter settings of SpecAugment (Park et al., ) include a time warping factor of 80, 2 frequency masking regions with a maximum width of 27, and 10 time masking regions with a maximum width of 100, along with a maximum masking fraction of 15% specifically for time masking. In our *CR-CTC* systems, we utilize larger amount of time masking through increasing both the number of time masking regions and the maximum masking fraction by a factor of 2.5.
Speed perturbation (Ko et al., ) with factors 0.9, 1.0 and 1.1 is applied to LibriSpeech (Panayotov et al., ) and Aishell-1 (Bu et al., ) datasets. The input features are 80-dimensional mel-spectrograms extracted using 25-ms windows with a 10-ms shift. For LibriSpeech and GigaSpeech datasets, we employ 500-class Byte Pair Encoding (BPE) (Sennrich et al., ) word pieces as modeling units, while for Aishell-1 dataset, we use 4336-class characters.
By default, we set $\alpha$ in Equation 3 to 0.2.
Zipformer (Yao et al., ), which uses dropout (Srivastava et al., ) and stochastic depth (Huang et al., ), is used as our speech encoder due to its speed and high performance.
Following (Yao et al., ), pruned transducer (Kuang et al., ), a highly optimized and memory-efficient version of transducer, is employed for comparison.
Word-error-rate (WER) and character-error-rate (CER) are employed as ASR metrics for English and Mandarin datasets, respectively.
As *CR-CTC* requires two forward pass during training, we train *CR-CTC* models with half the batch size and half the number of epochs compared to CTC models, ensuring a fair comparison in terms of training cost.
Training configuration in terms of number of GPUs and training epochs are provided in Appendix Section A.2.
For CTC and *CR-CTC* systems, we use prefix search decoding (Graves et al., ) with a beam size of 4 for comparisons against other state-of-the-art models, and employ greedy search decoding for ablation studies.
Results comparison between these two decoding methods are provided in Appendix Section A.3.

### 4.2 Comparison with state-of-the-art models

In this section, we compare our *CR-CTC* with other state-of-the-art models. For LibriSpeech and GigaSpeech datasets, we also use *CR-CTC* as an auxiliary loss in CTC/AED and pruned transducer systems for joint training (denoted as *CR-CTC*/AED and pruned transducer w/ *CR-CTC*), to further validate the representation learning capability of *CR-CTC*. For the larger GigaSpeech dataset, we additionally use a even larger scale of Zipformer (Zipformer-XL). Model configuration of different scales of Zipformer are provided in Table 14.
For Aishell-1 dataset, which is considerably smaller, we conduct experiments on Zipformer-S and Zipformer-M to ensure comparable parameter counts with other models reported in the literature.

LibriSpeech dataset. Table 1 presents the results on LibriSpeech dataset for *CR-CTC* and other state-of-the-art models. Our *CR-CTC* significantly outperforms the CTC baselines on all three scales of Zipformer encoder.
When comparing to CTC/AED models, our *CR-CTC* achieves lower WER on Zipformer-M/L, while yielding comparable result on Zipformer-S.
Similarly, our *CR-CTC* surpasses pruned transducer on Zipformer-M, and performs comparably on Zipformer-L.
It also demonstrates that *CR-CTC* can further enhance the performance of CTC/AED and pruned transducer models when used for jointly training.
A notable result is that pruned transducer combined with *CR-CTC* using Zipformer-L achieves a new state-of-the-art result of 1.88%/3.95% on test-clean/test-other, outperforming both the transducer models with Conformer-L (Gulati et al., ) and Stateformer 25L (Fathullah et al., ).

**Table 1: WER(%) performance of our method on LibriSpeech dataset compared to the best results reported in the literature without using an external language model.**
| Model | Params (M) | WER (%) |  |
| --- | --- | --- | --- |
| test-clean | test-other |  |  |
| CTC/AED, E-Branchformer-B (Kim et al., ) | 41.1 | 2.49 | 5.61 |
| CTC/AED, Branchformer (Peng et al., ) | 116.2 | 2.4 | 5.5 |
| CTC/AED, E-Branchformer-L (Kim et al., ) | 148.9 | 2.14 | 4.55 |
| Transducer, ContextNet-S (Han et al., ) | 10.8 | 2.9 | 7.0 |
| Transducer, ContextNet-M (Han et al., ) | 31.4 | 2.4 | 5.4 |
| Transducer, ContextNet-L (Han et al., ) | 112.7 | 2.1 | 4.6 |
| Transducer, Conformer-S (Gulati et al., ) | 10.3 | 2.7 | 6.3 |
| Transducer, Conformer-M (Gulati et al., ) | 30.7 | 2.3 | 5.0 |
| Transducer, Conformer-L (Gulati et al., ) | 118.8 | 2.1 | 4.3 |
| Transducer, MH-SSM 32L (Fathullah et al., ) | 140.3 | 2.01 | 4.61 |
| Transducer, Stateformer 25L (Fathullah et al., ) | 139.8 | 1.91 | 4.36 |
| CTC/AED, Zipformer-S (Yao et al., ) | 46.3 | 2.46 | 6.04 |
| CTC/AED, Zipformer-M (Yao et al., ) | 90.0 | 2.22 | 4.97 |
| CTC/AED, Zipformer-L (Yao et al., ) | 174.3 | 2.09 | 4.59 |
| Pruned transducer, Zipformer-S (Yao et al., ) | 23.3 | 2.42 | 5.73 |
| Pruned transducer, Zipformer-M (Yao et al., ) | 65.6 | 2.21 | 4.79 |
| Pruned transducer, Zipformer-L (Yao et al., ) | 148.4 | 2.00 | 4.38 |
| CTC, Zipformer-S | 22.1 | 2.85 | 6.89 |
| CTC, Zipformer-M | 64.3 | 2.52 | 6.02 |
| CTC, Zipformer-L | 147.0 | 2.5 | 5.72 |
| *CR-CTC*, Zipformer-S (ours) | 22.1 | 2.52 | 5.85 |
| *CR-CTC*, Zipformer-M (ours) | 64.3 | 2.1 | 4.61 |
| *CR-CTC*, Zipformer-L (ours) | 147.0 | 2.02 | 4.35 |
| *CR-CTC*/AED, Zipformer-L (ours) | 174.3 | 1.96 | 4.08 |
| Pruned transducer w/ *CR-CTC*, Zipformer-L (ours) | 148.8 | 1.88 | 3.95 |

Aishell-1 dataset. Table 2 presents the results on Aishell-1 dataset. Our *CR-CTC* models not only significantly outperform vanilla CTC by a substantial margin but also achieve better results than all other CTC/AED and pruned transducer models. For example, *CR-CTC* with Zipformer-S surpasses CTC/AED with Zipformer-M while using much fewer parameters.

**Table 2: WER(%) performance of our method on Aishell-1 dataset compared to the best results reported in the literature without using an external language model.**
| Model | Params (M) | WER (%) |  |
| --- | --- | --- | --- |
| dev | test |  |  |
| CTC/AED, Conformer in ESPnet (Watanabe et al., ) | 46.2 | 4.5 | 4.9 |
| CTC/AED, Conformer in WeNet (Yao et al., ) | 46.3 | $-$ | 4.61 |
| CTC/AED, E-Branchformer in ESPnet (Watanabe et al., ) | 37.9 | 4.2 | 4.5 |
| CTC/AED, Branchformer (Peng et al., ) | 45.4 | 4.19 | 4.43 |
| Pruned transducer, Zipformer-S (Yao et al., ) | 30.2 | 4.4 | 4.67 |
| Pruned transducer, Zipformer-M (Yao et al., ) | 73.4 | 4.13 | 4.4 |
| CTC, Zipformer-S | 23.1 | 4.89 | 5.26 |
| CTC, Zipformer-M | 66.2 | 4.47 | 4.8 |
| CTC/AED, Zipformer-S | 39.3 | 4.47 | 4.8 |
| CTC/AED, Zipformer-M | 83.2 | 4.0 | 4.32 |
| *CR-CTC*, Zipformer-S (ours) | 23.1 | 3.9 | 4.12 |
| *CR-CTC*, Zipformer-M (ours) | 66.2 | 3.72 | 4.02 |

GigaSpeech dataset. Table 3 shows the results on GigaSpeech dataset. Our *CR-CTC* consistently achieves a significantly lower WER than vanilla CTC across all scales of Zipformer.
In comparisons with CTC/AED or pruned transducer models, our *CR-CTC* demonstrates comparable performance on Zipformer L/XL.
Additionally, the results indicate that employing *CR-CTC* for joint training can further improve the performance of both CTC/AED and pruned transducer models.

**Table 3: WER(%) performance of our method on GigaSpeeech dataset compared to the best results reported in the literature without using an external language model.**
| Model | Params (M) | WER (%) |  |
| --- | --- | --- | --- |
| dev | test |  |  |
| CTC/AED, Transformer (Chen et al., ) | 87 | 12.30 | 12.30 |
| CTC/AED, Conformer in Wenet (Zhang et al., ) | 113.2 | 10.7 | 10.6 |
| CTC/AED, Conformer in ESPnet (Chen et al., ) | 113.2 | 10.9 | 10.8 |
| CTC/AED, E-Branchformer in ESPnet (Watanabe et al., ) | 148.9 | 10.6 | 10.5 |
| CTC, Zipformer-S | 22.1 | 12.08 | 11.95 |
| CTC, Zipformer-M | 64.3 | 11.23 | 11.27 |
| CTC, Zipformer-L | 147.0 | 11.16 | 11.16 |
| CTC, Zipformer-XL | 286.6 | 10.8 | 10.87 |
| CTC/AED, Zipformer-S | 46.3 | 11.4 | 11.39 |
| CTC/AED, Zipformer-M | 90.0 | 10.57 | 10.61 |
| CTC/AED, Zipformer-L | 174.3 | 10.26 | 10.38 |
| CTC/AED, Zipformer-XL | 315.5 | 10.22 | 10.33 |
| Pruned transducer, Zipformer-S | 23.3 | 10.98 | 10.94 |
| Pruned transducer, Zipformer-M | 65.6 | 10.37 | 10.42 |
| Pruned transducer, Zipformer-L | 148.4 | 10.23 | 10.28 |
| Pruned transducer, Zipformer-XL | 288.2 | 10.09 | 10.2 |
| *CR-CTC*, Zipformer-S (ours) | 22.1 | 11.68 | 11.58 |
| *CR-CTC*, Zipformer-M (ours) | 64.3 | 10.62 | 10.72 |
| *CR-CTC*, Zipformer-L (ours) | 147.0 | 10.31 | 10.41 |
| *CR-CTC*, Zipformer-XL (ours) | 286.6 | 10.15 | 10.28 |
| *CR-CTC*/AED, Zipformer-XL (ours) | 315.5 | 9.92 | 10.07 |
| Pruned transducer w/ *CR-CTC*, Zipformer-XL (ours) | 286.6 | 9.95 | 10.03 |

### 4.3 Ablation studies

We now perform ablation studies on LibriSpeech dataset using Zipformer-M encoder to investigate the effect of each component in *CR-CTC* (Section 3.2), and to validate our explanations of its behaviors (Section 3.3). Results of tuning $\alpha$ in Equation 3 and the ratio used to increase the amount of time masking are presented in Table 15.

Self-distillation.
As presented in Table 4, both the lack of increased time masking and the absence of different augmented views lead to WER degradation, indicating the effectiveness of enhancing the input diversity between sub-models during self-distillation. Replacing $D_{\mathrm{KL}}$ with hard label-based cross-entropy (CE) function in $\mathcal{L}_{\mathrm{CR}}$ (Equation 4) results in a WER degradation of 0.02%/0.22% on test-clean/test-other. This suggests the advantage of using $D_{\mathrm{KL}}$ which enables a finer-grained self-distillation as it distills over the full CTC lattice, whereas the hard label CE-based method only distills the best alignment. When removing the *sg* operation in $\mathcal{L}_{\mathrm{CR}}$, the WER increase by 0.12%/0.35%, which implies that the model might have a tendency towards a degenerated solution (Chen & He, ) that is insensitive to the pattern of input masking and model dropout.

**Table 4: Ablation studies for self-distillation in *CR-CTC* on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding.**
| Method | WER (%) |  |
| --- | --- | --- |
| test-clean | test-other |  |
| CTC baseline | 2.51 | 6.02 |
| *CR-CTC* (final) | 2.12 | 4.62 |
| No larger time masking | 2.19 | 4.98 |
| No larger time masking, no different augmented views | 2.27 | 5.11 |
| Use hard-label CE-based $\mathcal{L}_{\mathrm{CR}}$ | 2.14 | 4.84 |
| Remove *sg* in $\mathcal{L}_{\mathrm{CR}}$ | 2.24 | 4.97 |

Masked prediction. As reported in Table 5, without increasing the amount of time masking, the WER of *CR-CTC* increases by 0.07%/0.36% on test-clean/test-other, suggesting the effectiveness of enhancing the masked prediction behavior for contextual representation learning. Additionally, without using different augmented views, the WER increases further by 0.12%/0.13%. This indicates the advantage of independently applying random time masking, which improves the quality of the provided target distributions for the masked positions.
However, using larger amount of frequency masking leads to a WER degradation of 0.07% on test-clean, implying that the performance gain from increasing the amount of time masking is primarily due to the masked prediction behavior, rather than merely increasing the input diversity for the two branches.
Furthermore, applying larger amount of time masking does not benefit the CTC baseline, as it increases the WER by 0.17%/0.26%.
In the final *CR-CTC* system, excluding frames with time-masked regions in the current branch (self-masked) from $\mathcal{L}_{\mathrm{CR}}$ (Equation 4) leads to a larger WER degradation compared to excluding the remaining unmasked frames (self-unmasked). This highlights the importance of the masked prediction behavior in the overall performance of *CR-CTC*.

**Table 5: Ablation studies for masked prediction in *CR-CTC* on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding.**
| Method | WER (%) |  |
| --- | --- | --- |
| test-clean | test-other |  |
| CTC baseline | 2.51 | 6.02 |
| Use larger time masking | 2.68 | 6.28 |
| *CR-CTC* (final) | 2.12 | 4.62 |
| No larger time masking | 2.19 | 4.98 |
| No larger time masking, no different augmented views | 2.27 | 5.11 |
| No larger time masking, use larger frequency masking | 2.26 | 4.98 |
| Exclude self-masked frames in $\mathcal{L}_{\mathrm{CR}}$ | 2.32 | 5.26 |
| Exclude self-unmasked frames in $\mathcal{L}_{\mathrm{CR}}$ | 2.32 | 5.02 |

Peak suppression.
To measure the peakedness of the learned CTC distributions, we compute the averaged duration over all non-blank tokens, as well as the averaged emitting probabilities for the blank token and all non-blank tokens, based on the best alignment obtained through greedy search decoding on the test sets.
We also include the method *SR-CTC* (described in Appendix Section A.1) for comparison.
As presented in Table 6,
compared to the CTC baseline, *CR-CTC* learns smoother distributions and significantly improves the recognition performance.
Note that *SR-CTC* also surpasses the CTC baseline by 0.19%/0.8% on test-clean/test-other, while exhibiting a notably larger average duration of non-blank tokens.
This manifests the effectiveness of peak suppression in reducing overfitting and improving generalization performance.

**Table 6: Ablation studies for peak suppression in *CR-CTC* on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding. We include the averaged duration of all non-blank tokens, as well as the averaged emitting probabilities of the blank token and all non-blank tokens on the best alignments.**
| Method | Non-blank duration | Emit probability (%) | WER (%) |  |  |
| --- | --- | --- | --- | --- | --- |
| (frames) | blank | non-blank | test-clean | test-other |  |
| CTC baseline | 1.04 | 99.64 | 98.50 | 2.51 | 6.02 |
| *SR-CTC* | 4.25 | 95.44 | 90.04 | 2.32 | 5.22 |
| *CR-CTC* | 1.28 | 94.19 | 89.42 | 2.12 | 4.62 |

Compared to using auxiliary head for jointly training. The straightforward approach to improve the CTC performance is using an auxiliary head of AED (Chan et al., ; Watanabe et al., ) or pruned transducer (Kuang et al., ) for jointly training. As reported in Table 7, *CR-CTC* significantly outperforms these two methods with less model parameters, suggesting the advantage of our method.

**Table 7: Comparison between *CR-CTC* and methods using an auxiliary head for jointly training on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding.**
| Method | Params (M) | WER (%) |  |
| --- | --- | --- | --- |
| test-clean | test-other |  |  |
| CTC baseline | 64.3 | 2.51 | 6.02 |
| CTC w/ AED head | 90.0 | 2.46 | 5.57 |
| CTC w/ pruned transducer head | 65.8 | 2.42 | 5.4 |
| *CR-CTC* | 64.3 | 2.12 | 4.62 |

## 5 conclusion

In this work, we introduce the *CR-CTC* to enhance CTC performance. Specifically, it takes as input two different augmented views of the same speech mel-spectrogram, and enforce consistency between the two obtained CTC distributions. We explain our method from three different perspectives: 1) self-distillation between randomly sampled sub-models; 2) masked prediction for positions within time-masked regions, facilitating the learning of contextual representation; 3) peak suppression, which reduces overfitting and improves the model’s generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of *CR-CTC*. Additionally, detailed ablation studies validate our explanations.

## Appendix A Appendix

### A.1 Smooth-regularized CTC

Smooth-regularized CTC (*SR-CTC*) discourages peaky distributions by adding an smooth regularization loss (denoted as $\mathcal{L}_{\mathrm{SR}}$) to regular CTC model.
Specifically, we first apply a smooth kernel $K$ of size 3 to the model prediction $\mathbf{z}$, smoothing it along the time dimension: $\mathbf{z}^{(s)}=\mathrm{\emph{smooth}}(\mathbf{z},K)$ (^3^33The smoothing operation is done by using a 1-D depth-wise convolution layer.).
Then we minimize the $D_{\mathrm{KL}}$ between $\mathbf{z}$ and $\mathbf{z}^{(s)}$, similar to the consistency loss in *CR-CTC* (Equation 4):

$$ $\mathcal{L}_{\mathrm{SR}}(\mathbf{z},\mathbf{z}^{(s)})=\sum_{t=1}^{T}D_{ \mathrm{KL}}(\mathrm{\emph{sg}}(z^{(s)}_{t})\|z_{t}).$ (5) $$

The overall loss of *SR-CTC* is formulated as:

$$ $\mathcal{L}^{\prime}=\mathcal{L}_{\mathrm{CTC}}(\mathbf{x},\mathbf{y})+\beta \mathcal{L}_{\mathrm{SR}}(\mathbf{z},\mathbf{z}^{(s)}),$ (6) $$

where $\beta$ is a hyper-parameter. In this work, we use $K=(0.25,0.5,0.25)$ and $\beta=0.2$.

We validate its effectiveness in Section 4.3. Table 6 presents the experimental results.

### A.2 Training configuration

Training configuration, including the number of GPUs and training epochs, on LibriSpeech, Aishell-1 and GigaSpeech datasets are presented in Table 8, Table 9, and Table 10, respectively.

**Table 8: Training configuration on LibriSpeech dataset.**
| Model | GPUs | Epochs |
| --- | --- | --- |
| (80G NVIDIA Tesla A100) |  |  |
| CTC, Zipformer-S | 1 | 100 |
| CTC, Zipformer-M | 2 | 100 |
| CTC, Zipformer-L | 2 | 100 |
| *CR-CTC*, Zipformer-S | 1 | 50 |
| *CR-CTC*, Zipformer-M | 2 | 50 |
| *CR-CTC*, Zipformer-L | 2 | 50 |
| *CR-CTC*/AED, Zipformer-L | 2 | 50 |
| Pruned transducer w/ *CR-CTC*, Zipformer-L | 2 | 50 |

**Table 9: Training configuration on Aishell-1 dataset.**
| Model | GPUs | Epochs |
| --- | --- | --- |
| (80G NVIDIA Tesla A100) |  |  |
| CTC, Zipformer-S | 1 | 120 |
| CTC, Zipformer-M | 1 | 120 |
| CTC/AED, Zipformer-S | 1 | 60 |
| CTC/AED, Zipformer-M | 1 | 60 |
| *CR-CTC*, Zipformer-S | 1 | 60 |
| *CR-CTC*, Zipformer-M | 1 | 60 |

**Table 10: Training configuration on GigaSpeech dataset.**
| Model | GPUs | Epochs |
| --- | --- | --- |
| (80G NVIDIA Tesla A100) |  |  |
| CTC, Zipformer-S | 2 | 60 |
| CTC, Zipformer-M | 2 | 60 |
| CTC, Zipformer-L | 2 | 60 |
| CTC, Zipformer-XL | 4 | 60 |
| CTC/AED, Zipformer-S | 2 | 30 |
| CTC/AED, Zipformer-M | 2 | 30 |
| CTC/AED, Zipformer-L | 2 | 30 |
| CTC/AED, Zipformer-XL | 4 | 30 |
| Pruned transducer, Zipformer-S | 2 | 30 |
| Pruned transducer, Zipformer-M | 2 | 30 |
| Pruned transducer, Zipformer-L | 2 | 30 |
| Pruned transducer, Zipformer-XL | 4 | 30 |
| *CR-CTC*, Zipformer-S | 2 | 30 |
| *CR-CTC*, Zipformer-M | 2 | 30 |
| *CR-CTC*, Zipformer-L | 2 | 30 |
| *CR-CTC*, Zipformer-XL | 4 | 30 |
| *CR-CTC*/AED, Zipformer-XL | 4 | 30 |
| Pruned transducer w/ *CR-CTC*, Zipformer-XL | 4 | 30 |

### A.3 Results of different decoding methods

Results comparison between greedy search decoding and prefix search decoding for CTC and *CR-CTC* on LibriSpeech, Aishell-1 and GigaSpeech datasets are presented in Table 11, Table 12, and Table 13, respectively.

**Table 11: WER (%) results of different decoding methods on LibriSpeech dataset.**
| Model | Greedy search decoding | Prefix search decoding |  |  |
| --- | --- | --- | --- | --- |
| test-clean | test-other | test-clean | test-other |  |
| CTC, Zipformer-S | 2.85 | 6.91 | 2.85 | 6.89 |
| CTC, Zipformer-M | 2.51 | 6.02 | 2.52 | 6.02 |
| CTC, Zipformer-L | 2.49 | 5.7 | 2.5 | 5.72 |
| *CR-CTC*, Zipformer-S | 2.57 | 5.95 | 2.52 | 5.85 |
| *CR-CTC*, Zipformer-M | 2.12 | 4.62 | 2.1 | 4.61 |
| *CR-CTC*, Zipformer-L | 2.03 | 4.37 | 2.02 | 4.35 |

**Table 12: WER (%) results of different decoding methods on Aishell-1 dataset.**
| Model | Greedy search decoding | Prefix search decoding |  |  |
| --- | --- | --- | --- | --- |
| dev | test | dev | test |  |
| CTC, Zipformer-S | 4.88 | 5.26 | 4.89 | 5.26 |
| CTC, Zipformer-M | 4.46 | 4.8 | 4.47 | 4.8 |
| *CR-CTC*, Zipformer-S | 3.9 | 4.12 | 3.9 | 4.12 |
| *CR-CTC*, Zipformer-M | 3.73 | 4.02 | 3.72 | 4.02 |

**Table 13: WER (%) results of different decoding methods on GigaSpeech dataset.**
| Model | Greedy search decoding | Prefix search decoding |  |  |
| --- | --- | --- | --- | --- |
| dev | test | dev | test |  |
| CTC, Zipformer-S | 12.15 | 12.03 | 12.08 | 11.95 |
| CTC, Zipformer-M | 11.3 | 11.31 | 11.23 | 11.27 |
| CTC, Zipformer-L | 11.21 | 11.19 | 11.16 | 11.16 |
| CTC, Zipformer-XL | 10.85 | 10.91 | 10.8 | 10.87 |
| *CR-CTC*, Zipformer-S | 11.85 | 11.8 | 11.68 | 11.58 |
| *CR-CTC*, Zipformer-M | 10.78 | 10.88 | 10.62 | 10.72 |
| *CR-CTC*, Zipformer-L | 10.42 | 10.56 | 10.31 | 10.41 |
| *CR-CTC*, Zipformer-XL | 10.28 | 10.41 | 10.15 | 10.28 |

### A.4 Model configuration of different scales of Zipformer

Table 14 presents model configuration of different scales of Zipformer.

**Table 14: Model configuration of Zipformer at four different scales.**
| Scale | layer-numbers | embedding-dimensions | feed-forward-dimensions |
| --- | --- | --- | --- |
| S | {2,2,2,2,2,2} | {192,256,256,256,256,256} | {512,768,768,768,768,768} |
| M | {2,2,3,4,3,2} | {192,256,384,512,384,256} | {512,768,1024,1536,1024,768} |
| L | {2,2,4,5,4,2} | {192,256,512,768,512,256} | {512,768,1536,2048,1536,768} |
| XL | {2,2,4,5,4,2} | {192,384,768,1024,768,384} | {512,1024,2048,3072,2048,1024} |

### A.5 Ablation Studies on Hyper-parameter Tuning

Table 15 presents results of tuning hyper-parameters, including $\alpha$ in Equation 3 and the ratio used to increase the amount of time masking for *CR-CTC*.

**Table 15: Results of tuning $\alpha$ that controls $\mathcal{L}_{\mathrm{CR}}$ (Equation 3) and the ratio used to increase the amount of time-masking for *CR-CTC* on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding.**
| Hyper-parameter | WER (%) |  |
| --- | --- | --- |
| test-clean | test-other |  |
| $\alpha=0.1$ | 2.19 | 4.8 |
| $\alpha=0.2$ (final) | 2.12 | 4.62 |
| $\alpha=0.3$ | 2.23 | 4.84 |
| $1.0\times$ time masking | 2.19 | 4.98 |
| $1.5\times$ time masking | 2.19 | 4.73 |
| $2.0\times$ time masking | 2.17 | 4.71 |
| $2.5\times$ time masking (final) | 2.12 | 4.62 |
| $3.0\times$ time masking | 2.17 | 4.81 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: Please properly cite literature work and make proper comparison.
soundness: 3
strengths: The proposed idea is intuitive. Although the gain is small, the paper is compared with couple competitive baseline on librispeech.
summary: This paper applies a new type of self-consistency loss on different augmented view for CTC based ASR model. The new consistency regularized loss is doing KL over CTC output. Experimental results shows WER got improved.
weaknesses: The paper lack of citation to many very relevant work: 

Most important one:
Contrastive siamese network for semi-supervised speech recognition (https://arxiv.org/pdf/2205.14054). The paper focus on compare with SoTA, instead of compare with literature self-distill or other consistency based baseline. In that paper, it include practical trick to make SimSiam type of model work for ASR.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: For the combination of transducer and CR-CTC model, does the CTC score is used during the decoding?
soundness: 3
strengths: CR-CTC takes two different augmented views of the same speech mel-spectrogram as the inputs and enforce consistency between the two obtained CTC distributions. This method helps the model to do self-distillation between randomly sampled sub-models, learn contextual representation through masked prediction and reduce the peaky CTC distribution. The idea is simple and easy to implement. The training cost didn’t increase based on the description in section 4.1. The experiments on LibriSpeech, Aishell-1, and GigaSpeech show the proposed method outperform standard CTC in all these sets for different model sizes. It achieves comparable accuracy or even better than the advanced transducer or CTC/AED model. The paper also provided detailed ablation study to help the reader understand more details about the method.
summary: This paper proposes an improved version of Connectionist Temporal Classification (CTC) called Consistency-Regularized CTC (CR-CTC) for automatic speech recognition (ASR). CR-CTC enforces consistency between two CTC distributions obtained from different augmented views of the input speech mel-spectrogram.  The proposed method has 3 advantages: 1) it conducts self-distillation between random pairs of sub-models that process different augmented views; 2) it learns contextual representation through masked prediction for positions within time-masked regions, especially when we increase the amount of time masking; 3) it suppresses the extremely peaky CTC distributions, thereby reducing overfitting and improving the generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of CR-CTC, which achieves performance comparable to, or even slightly better than, that of transducer and CTC/AED.
weaknesses: Comparing results in table 1 and 3, the advantages of CR-CTC over standard CTC is smaller for GigaSpeech set than that for LibriSpeech set.  This may indicate that the proposed method may not work very well for big training data, e.g. tens of thousands of speech hours. 
    In table 1, 2 and 3, it’s not mentioned that the results for transducer and CTC/AED models are from beam search or greedy search. For these two models, the results of beam or greedy search usually have big differences. If the results given are from greedy search, it may mean the accuracy of CR-CTC model still have gap from that of transducer and CTC/AED model with beam search.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: Abstract starts a bit strange. It says CTC is worse than RNN-T and AED. Yes sure, we know. But then it talks about some method to maybe improve CTC. So why is mentioning RNN-T/AED relevant? Is it because you think the gap between CTC and AED/RNN-T is larger than what you would expect, and some methods like the presented here should close the gap? But I don't think that this really is being shown here in this work. Also, some variant of this method could maybe be applied to AED/RNN-T just as well. So, I don't really see why mentioning AED/RNN-T in the abstract is really relevant for this work here. It's fine in the introduction, to put CTC into perspective, but I don't think it's relevant in the abstract. I was a bit confused about this.

Eq 3 and also Figure 1, the CTC loss is maybe better formulated on z, not on x? I found it weird that x goes into L_CTC but z goes into L_CR.


"a time warping factor of 80" - what does that mean? I don't think you make the sequence 80 times longer?

Please clarify the downsampling of the Zipformer. Do you stick to the original Zipformer here, where the Conv Frontend downsamples the 100Hz feature frames to 50Hz, and then the residual/bypass connection is always at 50Hz, and at the very end, you downsample again to get 25Hz output frames, i.e. the log probs are at 25 Hz?

Did you investigate how the downsampling influences the CR-CTC loss? I think this can have quite a crucial impact, as we know that in general, for CTC, the ratio of input length to output length plays an important role for convergence and training dynamics.

Did you also test with other vocab sizes? 500 BPE size seems quite small to me. How does it perform with larger vocab sizes, e.g. with 10k?

How does the Zipformer influence the results? Specifically, do you think you get the same improvements with a normal Conformer?


"auxiliary head of AED" (p9, l483) (and also same with transducer) / Table 7: I don't exactly understand what you report here. Is the AED (or transducer) head just used as an aux loss, and during recognition, you only use the CTC head and ignore the AED (or transducer)? Please be more clear about that. Also, you are giving the wrong citation for that. The reference you give is about joint AED/CTC, where both AED and CTC heads are used for recognition, so nothing is ignored, nothing is just used as aux loss. The only reference I know where AED is used as an aux loss for CTC is "Keep Decoding Parallel With Effective Knowledge Distillation From Language Models To End-To-End Speech Recognisers", Hentschel et al, 2024.


On GigaSpeech, improvement seems much less (XL, test: 10.87 -> 10.28) compared to Librispeech (5.72 -> 4.35). Why?

Table 3 caption: "GigaSpeeech" typo.


Transducer w/ CR-CTC, what exactly is that? The same approach applied on transducer? But then this is not CTC? Or is it combined CTC with transducer?


Note, as your method is very simple to implement, and your improvements here are really impressive, I was just trying it out myself. However, with negative result: For my Conformer CTC baseline, on 100Hz inputs, downsampled by factor 6, with BPE 10k vocab, with aux AED loss ("Keep Decoding Parallel With Effective Knowledge Distillation From Language Models To End-To-End Speech Recognisers", Hentschel et al, 2024), where my baseline with greedy decoding without LM was at 5.93% on dev-other, it degraded with CR-CTC to 5.99% on dev-other. I halved the number epochs and halved the batch size for the CR experiment, just like you did. This is with CR loss scale 0.2. I did not adapt SpecAugment yet, but from your paper, I would expect that even with this setting, I should already see quite some improvement. So, why don't I? Your paper is lacking such study on other settings, as mentioned above (Conformer, other BPE sizes, other downsampling) to know whether I can/should expect similar improvements there or not, and whether I maybe need a very different CR loss scale there, or whether I need to care about other things.


Note, halving the batch size can have other effects. Many methods (e.g. optimizer, LR schedule, regularization, etc) don't work in the same way for different batch sizes. You do effectively more updates to the model. It can also have a regularization effect. So, I think an important missing experiment is: What happens to the baseline when you half the batch size? Maybe you also get improvements there?
soundness: 3
strengths: - Simple method.

- Seems to give huge improvements, at least in some settings.
summary: Consistency regularization (CR) is a well established existing method, where you forward through some model two times with different augmentation (and maybe also dropout or other randomness) to get two predictions, and then you minimize the symmetric KL between both, or similar. Thus, this method is purely on the training side, and doesn't change any modeling aspect.

Here, CR is applied to speech recognition, specifically to CTC models, on a frame-by-frame basis, called CR-CTC. The main difference in the two branches is caused by different SpecAugment masking.

For fair comparison, due to forwarding the data twice now, the number of epochs and the batch size are both halfed for the CR-CTC case.

Experiments are done on Librispeech, Gigaspeech and Aishell.

The method is mainly tested on CTC, but then some extension to that is when they used a joint AED/CTC model in the end, where CR is applied only to the CTC part.

A number of ablations has been made on the loss scale and on increasing the SpecAugment masking, which seems ot help more on CR-CTC, but for pure CTC, the original amount of SpecAugment masking already seems optimal.

The improvements on Librispeech test-other are quite large: From 5.72% WER to 4.35% WER.

It is also shown that it reduces the peakiness of the alignment behavior a bit.
weaknesses: - Some smaller details are a bit unclear.
- Only tested for Zipformer. More standard would be Conformer, but this is missing.
- Unclear how well this method works in other cases, e.g. other models, other datasets, some other hyperparams different. Specifically, I tested it in my setup, and it didn't really helped.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: * Why was the choice of alpha set to 0.2 in Equation 3? It would benefit readers if the authors could provide results from an ablation study showing the impact of different alpha values on performance. This would offer greater insight into the method's sensitivity to this hyperparameter choice.
* Why was a beam size of 4 specifically chosen for comparisons with other state-of-the-art models? The authors may consider including results with different beam sizes (e.g., 1, 4, 8) in an appendix to show the method's sensitivity to this parameter.
* The authors employed a larger amount of time-masking by increasing both the number of time-masking regions and the maximum masking fraction by a factor of 2.5. However, it would be interesting to know how much time-masking is optimal for masked prediction. The authors could provide results from an ablation study showing performance with different amounts of time-masking (e.g., 1x, 1.5x, 2x, 2.5x, 3x) for ZipformerXL and at least one baseline model in Table 15. This would help readers understand how critical this choice is and how generalizable the method is.
* In Tables 1, 2, and 3, are the CTC/AED baseline results reported with CTC-only decoding or CTC/AED joint decoding? This could be clarified by specifying the decoding methods for the baselines and the proposed method in the table captions.
* For a deeper understanding, the authors could include results showing the impact of increased time-masking on the baseline CTC model as well. This would help isolate whether the benefit comes from the two-branch architecture of CR-CTC or simply from more aggressive augmentation. Although the authors reported one of the baselines with larger time-masking, it would be helpful if results were provided for the other tables as well.
* In Table 3, the best results were obtained using Zipformer XL. However, the authors should:
1. Explain the rationale for using Zipformer-M in Tables 4 and 5 instead of Zipformer-XL.
2. Provide results for Zipformer-XL in Table 11 for completeness.
3. Clarify whether the results in Table 11 are from self-distillation or masked prediction in CR-CTC.
* The results of SR-CTC in Table 6 are slightly worse than those of CR-CTC. Do the authors have any explanation for this behavior, and does SR-CTC also use increased time-masking?
soundness: 3
strengths: * This paper presents a simple yet effective distillation method by extending time-masking to develop contextual representations from unmasked segments across two different augmented views.
* It demonstrates that CTC performance is comparable to that of transducer and CTC/AED models.
summary: This paper proposes a method to improve CTC performance by applying self-distillation between sub-models using drop-based techniques. The approach aims to enhance target token distribution predictions within time-masked regions and develop contextual representations from unmasked segments, drawing inspiration from self-supervised learning methods. By increasing time-masking, this technique promotes effective masked prediction, reducing peaky CTC distributions and strengthening the model's generalization ability. Experiments on multiple datasets—Librispeech, Aishell-I, and GigaSpeech—demonstrate that the proposed method achieves performance on par with transducer and CTC/AED models when used in joint training.
weaknesses: The method generates two different augmented views by independently applying existing SpecAugment to Zipformer. However, it raises the question of how generalizable this claim is when applied to other architectures like Conformer, E-Branchformer, or Branchformer.

Meta-review: This paper proposed to use consistency regulation to improve CTC training by enforcing constancy between two CTC distributions obtained from different augmented views of the input speech Mel-spectrogram.  The proposed method is simple but effective, showing impressive improvements from solid baseline on Librispeech, Aishell-1, and Gigaspeech. 

Reviewers questioned the novelty, scalability, and applicability of the method to other architectures beyond Zipformer. The authors rebutted by highlighting their focus on supervised training as opposed to self-supervised training in existing literature. They added more discussion of the related works, including missing references in the initial submission. To address architectural concerns, further experiments with Conformer showed similar improvements. Additionally, tests on the 10k-hour WenetSpeech dataset demonstrated convincing gains, alleviating scalability concerns.

Despite some reservations about the novelty, I concur with reviewers that the method is straightforward and effective. Considering the authors addressed reviewers’ concerns during rebuttal, the paper has the value for publication.

In summary, the strength of this paper is that it presents a simple but effective method to improve CTC performance by regularizing the constancy between two  CTC distributions obtained from different augmented views  of the input speech Mel-spectrogram. The weakness of the paper is that the consistency regularization has been studied in the literature (e.g., for self-supervised RNN-T training) although there is no work for supervised CTC training.

Decision: Accept (Poster)