CR-CTC: Consistency regularization on CTC for improved speech recognition
Authors: Zengwei Yao
Wei Kang
Xiaoyu Yang
Fangjun Kuang
Liyong Guo
Han Zhu
Zengrui Jin
Zhaoqing Li
Long Lin
Daniel Povey
Venue: ICLR 2025 Poster (2025)
Keywords: Consistency regularization
CTC
speech recognition
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR), renowned for its simplicity and computational efficiency. However, it often falls short in recognition performance.  In this work, we propose the Consistency-Regularized CTC (CR-CTC), which enforces consistency between two CTC distributions obtained from different augmented views of the input speech mel-spectrogram. We provide in-depth insights into its essential behaviors from three perspectives: 1) it conducts self-distillation between random pairs of sub-models that process different augmented views; 2) it learns contextual representation through masked prediction for positions within time-masked regions, especially when we increase the amount of time masking; 3) it suppresses the extremely peaky CTC distributions, thereby reducing overfitting and improving the generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of our CR-CTC. It significantly improves the CTC performance, achieving state-of-the-art results comparable to those attained by transducer or systems combining CTC and attention-based encoder-decoder (CTC/AED). We release our code at \url{https://github.com/k2-fsa/icefall}.

Published as a conference paper at ICLR 2025
CR-CTC: CONSISTENCY REGULARIZATION ON CTC FOR IMPROVED SPEECH RECOGNITION
Zengwei Yao, Wei Kang, Xiaoyu Yang, Fangjun Kuang, Liyong Guo, Han Zhu, Zengrui Jin, Zhaoqing Li, Long Lin, Daniel Povey Xiaomi Corp., Beijing, China dpoveyêx wwwiammii.cmmmm com
ABSTRACT
Connectionist Temporal Classification (CTC) is a widely used method for auto- matic speech recognition (ASR), renowned for its simplicity and computational efficiency.. However, it often falls short in recognition performance. In this work, we propose the Dassistenc-RReullaiin tency between two CTC distributions obtained from different augmented views of the input speech mel- mee-sppecroogam. We provide in-depth insights into its es- sential behaviors from three perspectives: 1) it conducts self-distillation between random pairs of sub-models that process different augmented views; 2) it learns contextual representation through masked prediction for positions within time- masked regions, especially when we increase the amount of time masking; 3) it suppresses the extremely peaky CTC distributions, thereby reducing overfit- ting and improving the generalization ability. Extensive experiments on Lib- riSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of our CR-CTC. It significantly improves the CTC performance, achieving state-of- the-art results comparable to those attained by transducer systems combining CTC and attention-based encoder-decoder (CTC/AED). We release our code at attps://github..
CTC (CR-CTC), which enforces consis-
INTRODUCTION
End-to-end approaches (Graves al. 2006 Graves 2012; Chan al. 2015). which eliminate the need of pre-aligned speech-text data, have replaced traditional hybrid systems (Bourlard & Mor- gan 2012, Hinton et al.) 2012) and become dominant methods in automatic speech recognition (ASR). Prominent examples include Connectionist Temporal Classification (CTC) (Graves et al., 2006). Transducer (Graves) 2012) (also known as RNN-T), and the method that combines CTC and attention-based encoder-decoder (AED) (Chan et al., 2015). referred as CTC/AED (Watan- abe et al. 2017). To handle the alignment between speech and token sequences, CTC Grrves al. 2006) introduces blank token and makes independent predictions each frame, training the model to maximize the total probability over all valid alignments. Transducer (Graves) 2012) extends CTC by introducing a prediction network and a joint network, explicitly modeling the interdependencies on output labels. CTC/AED (Watnnabe et al. (017) integrates CTC into AED (Chan et al. 2015) for jointly training, while the CTC and AED scores are fused during the decoding process. Among these three methods, CTC is the simplest and most computationally efficient due to its frame-independent assumption, making it strong candidate for real-world deployment. However, it significantly lags behind transducer and CTC/AED in terms of recognition performance, which limits its applicability.
To improve the CTC performance, in this work we propose the Munsistency-Regllariin CTC (CR- CTC), which takes two different augmented views of the same speech mel-spectrogram input, and enforces consistency between the resulting CTC distributions. We analyze its internal behaviors from three following perspectives. First, it performs self-distillation between sub-models randomly sampled by dase- techniques (Srivastava et al. 2014: Huang et al. 2016). Second, for positions within time-masked regions, the model is required to predict the target token distributions, forcing it to learn contextual representation based on unmasked context, akin to self-supervised learning methods (Devlin et al.) 2019: Baevski et al. 2020: Hsu et al., 2021). Therefore, we especially in- crease the amount of time masking in CR-CTC to enhance this masked prediction behavior. Third, the consistency regularization suppresses extremely peaky CTC distributions, which mitigates over-

---

Published as a conference paper at ICLR 2025
fitting and improves the model's generalization ability. Inspired by this, we additionally propose an simple method specifically designed to learn smoother CTC distributions (Appendix Section A.I
which is experimentally validated to be effective.
We conduct experiments on LibriSpeech, Aishell-1, I, and GigaSpeech datasets using Zipformer Yao et al. 2024) speech encoder. The results demonstrate the superiority of CR-C7C, which signif- icantly outperforms vanilla CTC and achieves results comparable to, or even slightly better than, those of transducer and CTC/AED. In addition, CR-CTC can further improve the performance of transducer and CTC/AED when employed for jointly training. We perform detailed ablation studies on LibriSpeech dataset to investigate the effect of each functional component in CR-CTC and to validate our explanations.
2 RELATED WORK
Self-distillation. Unlike traditional knowledge distillation (Buciluă al. 2006
Hinton et al. 2015),
which transfers knowledge from a larger and high-capacity teacher model to a smaller student model. self-distillation (Furlanello et al. 2018 Zhu al. [0118 Mobahi et al. 2020 Allen-Zhu & Li, 2020j involves learning from a same-architccture model that processes the same training data. This approach enables the model extract more refined representations and achieve improved perfor- mance. For example. BANs (Furlanello et al.) 2018) introduces a re-training procedure in which a newly initialized student model is trained to match a pre-trained teacher model, subsequently serv- ing as the teacher in the next iteration. Some works also explore constructing the teacher and student models from a shared network, distilling knowledge from deeper layers to shallower layers (Zhang et al. 2019: Kim et al., 2024), or between pairs of sub-models randomly initialized through drop- based techniques (Srivastava et al. 2014 Huang et al. 2016), such as R-Drop (Wu et al. 2021) and cosub (Touvron et al. (023). Our CR-CTC fundamentally conducts self-distill eeff-disiillatin between ran- dom sub-models, sharing similar idea to R-Drop and cosub, while our approach further use different augmented input views, which enriches the diversity of predictions from these sub-models.
Masked prediction has proven highly effective in self- eupfsuue learning De- vlin al.) [019]] Baevski et al. (219;; Doshi et al.] 2020 Baevski et al., [2000] HHsu al.) [021] He it al. 2022; Baevski et al., 2023). In this approach, the model is tasked with predicting masked positions based on the surrounding unmasked context, which encourages the learning of robust contextual representations. Notable methods for speech representation learning include wav2vec 2.0 (Baevski et al.] 2020). HuBERT et [ and data2vec 2.0 (Baevski et al.
Masked prediction.
(023). which
primarily differ in their prediction targets. Specifically, wav2vec 2.0 (Baevski et al. 2020) jointly
trains representation quantizer and learns to distinguish the true quantized target from distrac- tors (Oord et al.) 2018). HuBERT generates target labels through offline clustering, while data2vec 2.0 uses contextualized representations from a teacher model as its targets. Our CR-CTC essentially performs masked prediction for positions within time-masked regions, where the target labels are frame-level token distributions generated based on another augmented view of input.
Peaky CTC distributions. CTC models are known for predicting extremely peaky distribu- tions (Graves et al., 2006); Sak et al.) 2015). which can be harmful in certain scenarios, such as forced alignment (Huang al. 2024) and knowledge distillation (Ding al. 2020). These peaky distributions lead to inaccurate alignments as the model assigns excessive blanks to non-silence frames. To address this, label priors are employed to suppress the peaky distributions, thereby im- proving the accuracy of forced alignment (Huang et al., 2024). As position mismatches of CTC spikes can hinder knowledge distillation performance, some approaches propose encourage con- sistent alignments between the teacher and student (Ding et al. (020)) or to utilize sequence-level distillation (Takashima et al. 2019). Unlike previous works, we demonstrate that peak suppression in CR- DR-CTC can improve the generalization ability of the CTC models.
Consistency regularization. The technique of consistency regularization has demonstrated effec- tiveness in learning generalizable image representations across various learning paradigms, includ- ing self-supervised (Chen et al. 2020 Grill et al. 2020 He et al., 2020 Chen & He| 2021), semi- supervised (Sajaadi et al. 20066]
Laine & Aila 2016 Sohn et al. 2020), and supervised (Wu et al.) 2021 Touvron et al. 2023 Heo et al. 2023) learning tasks. Self sepervueeee methods, such as Sim- CLR Chen et al. 2020). B BOL (Grill et al. 2020), MoCo (He et al.) 2020) and SimSiam (Chen) & He 2021). aim to align hidden representations of unlabeled image data from different model branches or different augmented views. They address the training issue of feature collapsing into
2

---

Published as a conference paper at ICLR 2025
constant vector (Chen & He: 2021) through contrastive learning (Chen et al., 2020: He al. 2020
momentum encoder (Grill et al. 20200 He et al., 2020). and suop-gradiem operation (Chen He 2021). In semi-supervised learning. prominent example leveraging consistency regularization is FixMatch (Sohn et al. 2020). It generates pseudo-labels based on high-confidence predictions from weakly augmented images, then trains the model predict these pseudo-labels using the strongly augmented versions of the same images. Additionally, in supervised learning, methods such as R- Drop (Wu al. [222]) and cosub (Touvron et al., 2023) encourage consistency between predictions of randomly sampled sub-models using drop- based techniques.
When employing consistency regularization as unsupervised objective train transformer encoders on unlabeled speech data, a new training issue arises in the form of the shortcut learning prob- lem (Geirhos et al. 2020), which is tackled using reconstruction loss in Speech SimCLR (Jiang et al.) 2020) and temporal augmentation in C-Siam (Khorram et al., 2022). Some studies explore leverag- ing consistency regularization
enhance model robustness during predicting the pseudo-labels of untranscribed data, which are generated based on different augmentations (Masumura et al. 2020
Weninger et al. 2020 Chen et al. (0221)) Higuchi et al. 202T: ( 2022) or through speech chain reconstruction (Qi et al.] 2022). In contrast to these self/semi-superviseee supervised ASR works, our work fo- cuses on a fully supervised setting, where we introduce consistency loss as a regularization term improve performance of CTC model trained labeled data. As the consistency regularization is enforced on CTC distributions, which are stably supervised by the main CTC loss, it inherently avoids the training issues associated with the unsupervised objectives as observed in Speech Sim- CLR (Jiang et al. 2020) and C-Siam (Khorram et al. 20222..
The idea of R-Drop (Wu al. 2021 has also been extended to supervised ASR (Gao et al. 2022 Yoon et al. 2024). For example, to improve the CTC/AED system, (Gao et al. 2022) specially designs the spatial-temporal dropout construct the sub-models, with consistency regularization enforced exclusively on the CTC spike frames. Cons-KI KD (Yoon et al., (024) integrates consistency regularization into a knowledge distillation system, enabling the student model to be more robust to inconsistency induced by dropout. In this work, we focus on improving the performance of pure CTC systems and are the first to enable CTC models to match the performance of transducer and CTC/AED systems by simple yet effective approach. Moreover, we introduce peak suppression as a novel explanatory perspective, demonstrating for the first time that it can mitigate overfitting and enhance the generalization ability of CTC models.
3 METHOD
We first introduce the standard CTC algorithm in Section 3.1 plementation of our proposed Heristency-Reullaiizze CTC (CR-CC) in Section 3.2 followed by in-depth explanations from different perspectives in Section 3.3
Then we present the detailed im-
3.1 PRELIMINARY: CONNECTIONIST TEMPORAL CLASSIFICATION
The ASR task is to convert sequence of speech frames x {x,)T of length T to sequence of transcript tokens y = fy E D} of length U. where V is the vocabulary and typically T > U. CTC (Graves et al. 2006) extends the vocabulary V to V = VU {e) with blank token e, and aims to maximize the total posterior probability of all valid alignments π = { E V'HT between x and y. Let B(TT) denote the many-to-one map that merges repeating tokens and removes all blanks in n, and p(T) xx) denote the posterior probability of alignment ग, the CTC loss function is formulated as:
Lerc y) log
Σ p(T(x)).
(1)
TEB (v)
Specifically, given the input x, it employs encoder estimate the -d'l-dimnniional probability
distributions z = {ze 17: z = -f(x) where f is modeled by a speech encoder network such as Zipformer (Yao et al. 2024) followed by linear projection layer and softmax function. Note that we now start to use Lerc(z, y) instead of Lere(x, y) for ease of description in the following sections. Under the frame-indee traee-independet assumption Graves et al. 2006), p( x) is computed as:
p(T[x) = II 21,0Ta1
'Th is typically downsampled in the encoder f by factor of 4 for efficiency. This is omitted for the sake of simplicity in expression.
3

---

Published as a conference paper at ICLR 2025
Transcript y
Lerc(z(o,,) ,)
(CR(2())2(0))
(crc(z(o) .y
CTC output (
CTC output (0))
Share weights
Encoder f
Encoder
x
X
Augment
Augment
Mel-spectrogram
Figure Overall architecture CR-CTC.
where St, is the probability of emitting token ny at frame
3.2 OUR APPROACH: CONSISTENCY CTC
Figure illlustrates the overall architecture our proposed CR-CTC. takes as input two different augmented views, x(") and x()), both derived from the input speech mel-spectrogram x. The two input views are then passed through a shared speech encoder f. which estimates the per-frame distributions: 2aa) = f(x(a)), (b) = fx)()). In addition to computing the CTC losses on both branches: (cre(z(") y) and (Cr ,y), we introduce an auxiliary loss (defined in Equation 4h to enforce consistency between 2aa) and (b)); (z)(z!a) 20)) The overall loss of the whole model formulated as:
C= (CoTo(E
,) x)++ LCTo(z!)) y)) + (z) (0),
(3)
where a is hyper- hyper- parameter that controls the consistency regularization.
Different augmented views. The two different augmented views, x(a) and x()), are generated by in- dependently applying SpecAugment (Park al. (019) to two copies the input mel-spectrogram x. SpecAugment involves warping along time axis, masking blocks of frequency channels, and mask- ing blocks of time steps. Since time warping alters feature timing and thus shifts output timestamps, we apply it first before creating the copies to prevent significant timestamp mismatches between the outputs of two branches. Subsequently, random frequency masking and time masking are both applied the two copies, resulting in x()) and x()), Note that we also increase the amount of time masking by factor of 2.5 compared to regular systems. The reason behind this adjustment is explained in Section 3.3 with implementation details provided in Section 4.1
Consistency regularization loss. The consistency regularization is applied on each frame f, by min- imizing the bidirectional Kullback-Leible divergence (denoted as DKL. between each pair of distri- butions
2, and 2/h) (sL(sg(E (z(") (1(()) and (sg(s I|2"") where denotes the operation stopping gradient on the target distributions. The regularization loss LCR(Z (b)) is formulated as:
T (c)() = Σ t=1
(s (1)())))))))))))))))))))))))))) + (KI,(g(22)" (1==()))))) (1))
(4)
4

---

Published as a conference paper at ICLR 2025
3.3 EXPLANATION
We now explain the essential behaviors of our proposed CR-CTC CTC from three different perspectives: 1) it performs self-distillation between pairs of sub-models with different input views; 2) conducts contextual representation learning by predicting the token distributions at masked positions based on unmasked context; 3) it suppresses extremely peaky CTC distributions, mitigating overfitting and enhancing generalization ability. We conduct empirical investigation through ablation studies in Section and the experimental results validate our explanations.
Self-distillation. When using model regularization techniques such dropout (Srivastava et al. 2014) and stochastic depth (Huang et al., 2016), which randomly drop parts of the model (neurons or layers), it can be viewed as implicitly traiming randomly sampled sub-models that ultimately combined into an ensemble during inference. Similar to R-Drop (Wu et al., (021) and cosub [Tou-] vron al. 2023), in CR-CTC, enforcing consistency regularization between the two branches ables to perform self-distillation between pairs of randomly sampled sub-models derived from the shared model f. with each sub-model receiving supervision signals in the form of per-frame pre- dictions from the other. In addition, feeding different augmented views (with larger amount of time masking) exposes these sub-models to varied aspects of the input data, enhancing their prediction diversity and facilitating richer knowledge transfer as well as complementary representation learn- ing.
Masked prediction. In CR-CTC, consistency regularization requires frames within the time-masked regions each branch to predict the corresponding token distributions, which are generated by the other branch the fly. Similar to masked- self-supervised models (Devlin et al.) 2019 Baevski et al. 2020; Hsu et al. 202T]. this behavior encourages the model to capture acoustic in- formation on the unmasked context and exploit its implicit language modeling capability. Inde- pendently applying random time masking to the two branches reduces the occurrence of positions masked by both branches, thereby improve the quality of the provided target distributions for these masked positions. Furthermore, increasing the amount of time masking in CR-CTC enhances con- textual representation learning through the masked prediction behavior.
Peak suppression. In line with previous works (Graves et al. 2006 Sak et al. 2015). we also observe that CTC tends to learn extremely peaky distributions. As shown in Figure 2 (left) almost all non-blank tokens occupy only one frame, while the remaining frames are dominated by the blank token, with both types of emissions occurring with extremely high probabilities. This phenomenon suggests potential overfitting to training data, which limits generalization ability to unseen data.
Enforeing prediction consistency between the two branches in CR-CTC guides the model learn the average of their predictions, ultimately resulting in smoother distributions. The peak suppression behavior reduces overconfidence on training data, thereby improving the model's generalization ability. As presented in Figure (right), CR-CTC exhibits reduced token emitting probabilities and an increased occurrence of repeating non-blank tokens. A comparison of concrete statistics on the distribution peakedness between CTC and CR-CTC is provided in Table
Inspired by this, we also propose a simple method, called Emooth-Regulane which incorporates auxiliary loss into regular CTC, specifically encouraging the model to learn smoother CTC distributions. Appendix Section A. presents the details of SR-CTC.
CTC (SR-CTC),
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Datasets. To evaluate the effectiveness of our proposed CR-CTC, we conduct experiments on three publicly available ASR datasets: 1) LibriSpeech (Panayotov et al. 2015). which contains 1000 hours of English speech; 2) Aishell-1 (Bu et al. 2017). which consists of 170 hours of Mandarin speech; 2021a), comprising 10000 hours of English speech.
3) GigaSpeech (Chen et al.
Implementation details. Our experiments are performed using the icefall framework with Lhotse toolkit (Zelasko et al. 2021) for data preparation. For regular ASR recipes in icefall, default param- eter settings of SpecAugment (Park et al. 2019) include a time warping factor of 80, 2 frequency
htttp:://githbb.com
5

---

Published as a conference paper at ICLR 2025
probabiitty
1.0
Token probabbitty
1.0
0.5
0.5
11
0.0
0.0
20
40
Framess
Frames
(a) Sample in CTC
(h) Sample in CR-CTC
probabiitty
Tiken probability
THEN
WHATT 1
AND
WHATT 101
THEN
0
10
20
30
40
50
60
10
20
30
40
50
60
(c) Sample in CTC
(d) Sample 2 in CR-CTC
probability
1.0
Tiken probabiiity
1.0
0.5
0.5
THE
00 255 F FOLLDW
7ALUE 7ER
RA
75 FOLLDW kWW
0.0
0.0
10
50
Frames
Frames
(e) Sample in CTC
(f) Sample 3 CR-CTC
probabillty
Token probabiity
THAT
MOTHING
COPP
NOTHING
0
10
40
50
10
20
90
(g) Sample in CTC
(h) Sample 4 in CR-CTC
Figure 2: Visualization of token emitting probabilities for vanilla CTC (left) and our CR-CTC (right) on four randomly selected samples from LibriSpeech test set. The gray dashed lines indicate the blank token. Compared to vanilla CTC, the token distributions in CR-CTC are smoother with lower emitting probabilities and more repeating non-blank tokens.
masking regions with maximum width of 27, and 10 time masking regions with maximum width of 100, along with maximum masking fraction of 15% specifically for time masking] In our CR- CTC systems, we utilize larger amount time masking through increasing both the number of time masking regions and the maximum masking fraction by factor of 2.5. Speed perturbation (Ko et al.) 2015) with factors 0.9, 1.0 and
applied LibriSpeech (Panayotov et al.] (015) and Aishell- (Bu al., 2017) datasets. The input features are 80-dimensional mel-spectrograms extracted using 25-ms windows with a 10-ms shift. For LibriSpeech and GigaSpeech datasets, we employ 500-class Byte Pair Encoding (BPE) (Sennrich et al. 2016) word pieces modeling units, while for Aishell-1 dataset, we use 4336-class characters. By default, we set a in Equation 3 to 0.2. Zip- former (Yao et al.] (024), which uses dropout (Srivastava al. 2014) and stochastic depth (Huang jet al. 2016), is used as our speech encoder due to its speed and high performance. It takes input features frame of 100Hz, processes the sequence through 6 stacks with frame rates of 50Hz, 25Hz, 12.5Hz, 6.25Hz, 12.5Hz, and 25Hz, and finally produces the encoder output at frame rate of 25Hz. Following (Yao al.) 2024), pruned transducer (Kuang al.) (022). highly optimized and memory- efficiiet version of transducer, is employed for comparison. Word-error-rate (WER) and character-error-rate (CER) are employed as ASR metrics for English and Mandarin datasets, respectively. As CR-CTE FRCCTC requires two forward pass during training. we train CR- CR-CTC models with half batch size and half the number of epochs compared to CTC models, ensuring a fair com-
See the SpecAugment implementation in Lhotse for more details: httpa://gitthi com/ Thot hotefeeepppeee//oae/////aaaaa//e
trans w.ransforrs.pp forms.py

---

Published as a conference paper at ICLR 2025
parison in terms of training cost. Training configuration in terms of number of GPUs and training epochs are provided in Appendix Section A.2 For CTC and CR-CTC systems, we use prefix search decoding (Graves et al. 2006) with beam size of 4 for comparisons against other sate-of--hee-ad art models, and employ greedy search decoding for ablation studies. Results comparison between these two decoding methods are provided in Appendix Section A.3] For pruned transducer models, use beam search decoding with beam size of 4 (Kang et al. 2023). For CTC/AED systems, we use joint decoding that combines CTC scores and AED scores
Watanabe et al. 2017
4.2 COMPARISON WITH STATE-OF-TIE-ANE MODELS
In this section, compare CR-CTC with other state-of-the-art models. For LibriSpeech and GigaSpeech datasets, we also use CR-CI CR-CTC as an auxiliary loss in CTC/AED and pruned transducer systems for joint training (denoted as CR-CTC/AED and pruned transducer w/ CR-CTC), further validate the representation learning capability of CR-CTC. Note that for the models that combine CR-CTC and pruned transducer, we only utilize the transducer head for decoding, without incor- porating the CTC scores. For the larger GigaSpeech dataset, we additionally use even larger scale of Zipformer (Zipformer-XL), Model configuration of different scales of Zipformer are pro- vided in Table 13| For Aishell-1 dataset, which is considerably smaller, we conduct experiments on Zipformer-S and Zipformer-M ensure comparable parameter counts with other models reported in the literature.
Table I: WER(%) performance our method on LibriSpeech dataset compared to the best results reported in the literature without using external language model.
WER (%) test-other
Model
Params (M)
test-clean
CTC/AED, E-Branchformer-BB Kim
2023
41.1
2.49
5.61
CTC/AED, Branchformer (Peng
2022
116.2
2.4
5.5
CTC/AED, E-Branchformer-ll, Kim cl
2023
148.9
2.14
4.55
Transducer, ContextNet-S
Han et al.
2020
10.8
2.9
7.0
Transducer, ContextNet-M
(Han al.
2020
31.4
2.4
5.4
Transducer, ContextNet-L
Han et
2020
112.7
2.1
4.6
Transducer, Conformer S Gulati al.
2020
10.3
2.7
6.3
Transducer, ConformerMM
Guilat ct al.
2020
30.7
2.3
5.0
Transducer, Conformer-l. Gullti ct al.
2020
118.8
2.1
4.3
Transducer, MH-SSM 321. (Fatuullah
2023
140.3
2.01
4.61
Transducer, Statefommer (Falhullah
2023)
139.8
1.91
4.36
CTC/AED, Zipformer-S Yao et al. 2024
46.3
2.46
6.04
CIC/AED, Zipformer-M Yao 2024
90.0
2.22
4.97
CTC/AED. Zipformer-L Yao
2024)
174.3
2.09
4.59
Pruned transducer, Zipformer-S Yao el al.
2024
23.3
2.42
5.73
Pruned transducer, Ziplormer-M Yao cl al.
20244
65.6
2.21
4.79
Pruned transducer, Zipformer-L.
Yao et al. 2024
148.4
2.00
4.38
CTC. Zipformer-S
22.1
6.89
CTC. Zipformer-M
64.3
2.52
6.02
CTC, Zipformer-L.
147.0
2.5
5.72
CR-CTC Zipformer-S (ours)
22.1
2.52
5.85
CR- CRCC Z (ours)
64.3
2.1
4.61
CR-CTC, Zipformer-1. (ours)
147.0
2.02
4.35
CR- ER-CTC/AED, Zipformer-L (ours)
174.3
1.96
4.08
Pruned transducer w/ CR-CTC, ZipformerLL (ours)
148.8
1.88
3.95
LibriSpeech dataset. Table I presents the results on LibriSpeech dataset for CR- CR-CTC and other state-of-the-ard models. Our CR-CTC significantly outperforms the CTC baselines on all three scales of Zipformer encoder. When comparing CTC/AED models, our CR-CTC achieves lower WER on Zipformer-Mi M/L,, while yielding comparable result on Zipformer-S. Similarly, our CR-CTC surpasses pruned transducer Zipformer-M. and performs comparably that CR-CU CTC can further enhance the performance of CTС/AED and pruned transducer models when used for jointly training. A notable result is that pruned transducer combined with CR-CTC
Zipformer-L. It also demonstrates

---

Published as a conference paper at ICLR 2025
using Zipformer-L achieves new ofte--o-thhe-ar result of 1.88% test-clean/test-otiee outperforning both the transducer models with Conformer-L (Gulati et al. 2020) and Stateformer 25L (Fathullah al. 2023).
Table 2: WER(%) performance of our method on Aishell-1 dataset compared to the best results reported in the literature without using an external language model.
WER (%)
Model
Params (M)
dev
test
CTC/AED, Conformer in ESPnet Watanabe cl al.
2018
46.2
4.5
4.9
CTC/AED, Conformer in WeNet Yao el al. 202T
46.3
4.61
CTC/AED, E Branchformer in ESP'net Watanabe et al. 2018
37.9
4,2
4.5
CTC/AED, Branchformer (Peng et al. 2022)
45.4
4.19
4.43
Pruned transducer, Zipformer-S Yao et al.
2024 2024
30.2
4.67
Pruned transducer, Zipformer-M Yao et al.
73.4
4.13
4.4
CTC, Zipformer-S
23.1
4.89
5.26
CTC, Zipformer-M
66.2
4.47
4.8
CTC/AED, Zipformer-S
39.3
4.47
4.8
CTC/AED, Zipformer-M
83.2
4.0
4.32
CR-CTC, Zipformer-S (ours)
23.1
4.12
CR- CR-CTC, Zipformer-M (ours)
66.2
3.72
4.02
Aishell-1 dataset. Table 2 presents the results on Aishell-1 dataset. Our CR-CTC models not only significantly outperform vanilla CTC by substantial margin but also achieve better results than all other CTC/AED and pruned transducer models. For example, CR-CTC with Zipformer-S surpasses CTC/AED with Zipformer-M while using much fewer parameters.
Table 3: WER(%) performance of our method on GigaSpeech dataset compared to the best results reported in the literature without using an external language model.
WER (%)
Model
Params (M)
dev
test
CTC/AED, Transformer (Chen et al.
202
87
12.30
12.30
CTC/AED, Conformer in Wenel Zhang cl a.
2022
113.2
10.7
10.6
CTC/AED, Conformer in ESPnet (Chen cl a. 20218
113.2
10.9
10.8
CTC/AED, E-Branchformer in ESP Walanabe cl al. 2018
148.9
10.6
10.5
CTC, Zipformer-S
22.1
12.08
11.95
CTC, Zipformer-M
64.3
11.23
11.27
CTC, Zipformer-L
147.0
11.16
11.16
CTC, XiL
286.6
10.8
10.87
CTC/AED, Zipformer-S
46.3
11.4
11.39
CTC/AED, Zipformer-M
90.0
10.57
10.61
CTC/AED, Zipformer-L CTC/AED, Zipformer-XL.
174.3
10,26
10.38
315.5
10.22
10.33
Pruned transducer, Zipformer-S
23.3
10.98
10.94
Pruned transducer, Zipformer-M
65.6
10.37
10.42
Pruned transducer, Zipformer-l.
148.4
10.23
10.28
Pruned transducer, Zipformer-XI.
288.2
10.09
10.2
CR-CTC, Zipformer-S (ours)
22.1
11.68
11.58
CR-CTC, Zipformer-M (ours)
64.3
10.62
10.72
CR-CTC, Zipformer-L. (ours)
147.0
10.31
10.41
CR-CTC, Zipformer-XI (ours)
286.6
10.15
10.28
CR-CTC/AED, Zipformer-XL (ours)
315.5
9.92
10.07
Pruned transducer w/ CR-CTC, XX
286.6
9.95
10.03
GigaSpeech dataset. Table shows the results GigaSpeech dataset. Our CR-CTC consistently achieves a significantly lower WER than vanilla CTC across all scales of Zipformer. In comparisons with CTC/AED pruned transducer models, our CR-CTC demonstrates comparable performance
8

---

Published as a conference paper at ICLR 2025
on Zipformer L/XL. Additionally, the results indicate that employing CR- CR-CTC for joint training can further improve the performance of both CTC/AED and pruned transducer models.
4.3 ABLATION STUDIES
We now perform ablation studies LibriSpeech dataset using Zipformer-M encoder investigate the effect of each component in CR-CTC (Section (.2), and to validate our explanations of its be- haviors (Section 3.3). Results of tuning in Equation and the ratio used to increase the amount of time masking are presented in Table T6
Table 4: Ablation studies for celf-distillation in CR-CTC on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding.
WER (%)
Method
test-clean
test-other
CTC bascline
2.51
6.02
EMA-distilldd CTC
2.31
5.25
CR-CTC (final)
2.12
4.62
No larger time masking
2.19
4.98
No larger time masking, different augmented views
2.27
5.11
Use hare-label CE-based LCR
2.14
4.84
Remove sg in LCR
2.24
4.97
Self-distillation. One self-distillation method in self-supervised learning is construct teacher model by tracking the model weights using exponential moving average (EMA) (Grill et al. [020 He al. 2020; Baevski et al. 2023). For comparison,
include this approach, referred as EMA-distilled CTC, which incorporates an auxiliary loss to learn from the CTC distribution of the
EMA teacher model. Its details are provided in Appendix Section A.6 presented in Table CR-
CTC significantly outperforms EMA-distilled CTC, demonstrating its superiority in seff-dittillation For CR-CTC, both the lack of increased time masking and the absence of different augmented views lead to WER degradation, indicating the effectiveness of enhancing the input diversity between sub- models during self-distillation. Replacing DKL with hard label-based cross-entropy (CE) function in CCR (Equation 4 results in a WER degradation of 0.02% 70.2%% on test-cleanftest-other, This suggests the advantage of using DKL which enables finer aner-grained self- oeffdistillation as it distills over the full CTC lattice, whereas the hard label CE- method only distills the best alignment. When removing the sg operation in LCR, the WER increase by 0. 0.12%% 11%/0.35% which implies that the model might have tendency towards degenerated solution (Chen & He 2021 2 that is insensitive the
pattern of input masking and model dropout.
Table 5: Ablation studies for masked prediction in CR-CTC on LibriSpeech dataset using Zipformer- M encoder and greedy search decoding.
WER (%) test-other
Method
test-clean
CTC baseline
2.51
6.02
Use larger time masking
2.68
6.28
CR-CTC (final)
2.12
4.62
No larger time masking
2.19
4.98
No larger time masking, no different augmented views
2.27
5.11
No larger time masking, use larger frequeney masking
2.26
4.98
Exclude self-masked frames in LCR
2.32
5.26
Exclude self-unmasked frames in LCR
2.32
5.02
Masked prediction. As reported in Table 5 without increasing the amount of time masking, the WER of CR-CTC increases by 0.07% on rest-cleanttest-otheer suggesting the effectiveness of enhancing the masked prediction behavior for contextual representation learning. Additionally, without using different augmented views, the WER increases further by 0. 0.12% This indi- cates the advantage of independently applying random time masking, which improves the quality
9

---

Published as a conference paper at ICLR 2025
of the provided target distributions for the masked positions. However, using larger amount of frequency masking leads to a WER degradation of 0.07% on test-clean, implying that the perfor- mance gain from increasing the amount of time masking is primarily due to the masked prediction behavior, rather than merely increasing the input diversity for the two branches. Furthermore, ap- plying larger amount of time masking does not benefit the CTC baseline, it increases the WER by 0. 0.17%/0.26% In the final CR-CTC system, excluding frames with time-masked regions in the current branch (self-masked) from LCR (Equation 4 leads larger WER degradation compared to excluding the remaining unmasked frames (self- unfaseddd.. This highlights the importance of the masked prediction behavior in the overall performance of CR-CTC.
Table 6: Ablation studies for peak suppression in CR-CTC on LibriSpeech dataset using Zipformer- M encoder and greedy search decoding. We include the averaged duration of all non-blank tokens, as well the averaged emitting probabilities of the blank token and all non-blank tokens on the best alignments.
Non-blank duration (frames)
Emit probability (%)
WER (%)
Method
blank
non-blank
test-clean
test-other
CTC baseline
1.04
99.64
98.50
2.51
6.02
SR-CTC
4.25
95.44
90.04
2.32
5.22
CR-CTC
1.28
94.19
89.42
2.12
4.62
Peak suppression. measure the peakedness of the learned CTC distributions, we compute the averaged duration over all non-blank tokens, as well as the averaged emitting probabilities for the blank token and all non-blank tokens, based on the best alignment obtained through greedy search decoding the test sets. We also include the method SR-CTC (described in Appendix Section A.I] for comparison. As presented in Table [6] compared to the CTC baseline, CR-CTC learns smoother distributions and significantly improves the recognition performance. Note that SR-CTC also sur- passes the CTC baseline by 11%9/..8% on test-cleanitest-otle
while exhibiting a notably larger average duration of non-blank tokens. This manifests the effectiveness of peak suppression re- ducing overfitting and improving generalization performance.
Table 7: Comparison between CR-CTC and methods using an auxiliary head for jointly training on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding.
WER (%)
Method
Params (M)
test-clean
test-other
CTC baseline
64.3
2.51
6.02
CTC w/ AED head
90.0
2.46
5.57
CTC w/ pruned transducer head
65.8
2.42
5.4
CR-CTC
64.3
2.12
4.62
Compared to using auxiliary head for jointly training. The straightforward approach to improve the CTC performance using auxiliary head of AED (Chan al. [0055] Hentschel et al.) (024) or pruned transducer (Kuang et al. 2022h for jointly training, while retaining only the CTC head for inference. As reported in Table 7 CR- CTC significantly outperforms these two methods with less
model parameters, suggesting the advantage of our method.
CONCLUSION
In this work, we introduce the CR-CTC to enhance CTC performance. Specifically, it takes as input two different augmented views of the same speech mel-spectrogram, and enforce consis- tency between the two obtained CTC distributions. We explain our method from three different perspectives: 1) self-distillation between randomly sampled sub-models; 2) masked prediction for positions within time-masker regions, facilitating the learning of contextual representation: 3) peak suppression, which reduces overfitting and improves the model's generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of CR-CTC. Additionally, detailed ablation studies validate our explanations.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.098/6, 2020.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453, 2019.
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning speech representations. Advances in neural information processing systems, 33: 1 12449-12460, 2020.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Con- ference on Machine Learning, pp. 1416-1429. PMLR, 2023.
Herve A Bourlard and Nelson Morgan. Connectionist speech recognition: hybrid approach, vol- ume 247. Springer Science & Business Media, 2012.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA), pp. 1-5, 2017.
Cristian Buciluä, Rich Caruana, and Alexandru Niculescu-Mizi.. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv: 0/2/1 2015.
Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain ast corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106. 100.:09906.009, 2021a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Zhehuai Chen, Andrew Rosenberg, Yu Zhang, Heiga Zen, Mohammadreza Ghodsi, Yinghui Huang, Jesse Emond, Gary Wang, Bhuvana Ramabhadran, and Pedro J Moreno. Semi-supervision in asr: Sequential mixmatch and factorized uts-based augmentation. In Interspeech, pp. 736-740, 2021b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 11866. 2019.
Haisong Ding. Kai Chen, and Qiang Huo. Improving knowledge distillation of ctc-trained acoustic models with aligament-consisteet ensemble and target delay. IEEE/ACM transactions on audio, speech, and language processing, 28:2561-2571, 2020.
Yassir Fathullah, Chunyang Wu, Yuan Shangguan, Junteng Jia, Wenhan Xiong, Jay Mahadeokar, Chunxi Liu, Yangyang Shi, Ozlem Kalinli, Mike Seltzer, et al. Multi-head state space model for speech recognition. arXiv preprint arXiv:2305. 12498, 2023.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pp. 1607-1616. PMLR, 2018.
11

---

Published as a conference paper at ICLR 2025
Yingying Gao, Junlan Feng, Tianrui Wang. Chao Deng, and Shilei Zhang. A etc triggered siamese network with temporal dropout for speech recognition. arXiv preprint wrXiv:2206.0803, 2022.
Robert Geirhos,
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1)::66)66777 673, 2020.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv:1211.3711, 2012.
arXiv preprint
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Jean-Ba Man-BBstien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernard Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution Convolution-augmentee augmented Transformer for Speech Recognition. In Proc. Interspeech 2020, pp. 5036-5040, 2020.
Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang. and Yonghui Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. In Interspeech 2020, pp. 3610-3614, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Michael Hentschel, Yuta Nishikawa, Tatsuya Komatsu, and Yusuke Fujita. Keep decoding parallel with effective knowledge distillation from language models end-to-end speech recognisers. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10876 10880. IEEE, 2024.
Byeongho Heo, Taekyung Kim, Sangdoo Yun, and Dongyoon Han. Masking augmentation for supervised learning, arXiv preprint arXiv:2306. 11339, 2023.
Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori Momentum pseudo-labeling for supi-supe speech recognition. arXiv preprint arXiv:2106.0892. 08922, 2021.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. 1EEE Signal processing magazine, 29(6):2299) 97, 2012.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM transactions on audio, speech, and language processing, 29:3451-3461 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14 2016, Proceedings, Part IV 14, pp. 646-661, 2016.
12

---

Published as a conference paper at ICLR 2025
Ruizhe Huang, Xiaohui Zhang, Zhaoheng Ni, Li Sun, Moto Hira, Jeff Hwang, Vimal Manohar, Vineel Pratap, Matthew Wiesner, Shinji Watanabe, et al. Less peaky and more accurate ctc forced alignment by label priors. In ICASSP 2024-2024 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 11831-11835. IEFF, 2024.
Dongwei Jiang. Wubo Li, Miao Cao, Wei Zou, and Xiangang Li. Speech simelr: Combining con- trastive and reconstruction objective for self-supervised speech representation learning. arXiv preprint arXiv:2010.1330000
2020.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64-77, 2020.
Wei Kang, Liyong Guo, Fangjun Kuang, Long Lin, Mingshuang Luo, Zengwei Yao, Xiaoyu Yang, Piotr Zelasko, and Daniel Povey. Fast and parallel decoding for transducer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Soheil Khorram, Jaeyoung Kim, Anshuman Tripathi, Han Lu, Qian Zhang, and Hasim Sak. Con- trastive siamese network for semi-supervised speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7207-7211. IEEE, 2022.
Eungbeom Kim, Hantae Kim, and Kyogu Lee. Guiding frame-level ctc alignments using self- knowledge distillation. arXiv preprint arXiv:2406.07909 2024.
Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu Han, and Shinji Watan- abe. E-branchformer: Branchformer with enhanced merging for speech recognition. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 91 IEEE, 2023.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. Audio augmentation for speech recognition. In Sixteenth annual conference of the international speech communication
association, 2015.
Fangjun Kuang, Liyong Guo, Wei Kang, Long Lin, Mingshuang Luo, Zengwei Yao, and Daniel Povey. Pruned rn-tt for fast, memory-efficient asr training. arXiv preprint arXiv:2206.133: 13236, 2022.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 1el.:266000224,, 2016.
Ryo Masumura, Mana Ihori, Akihiko Takashima, Takafumi Moriya, Atsushi Ando, and Yusuke Shinohara. Sequence-level consistency training for semi-supervised end-to-end automatic speech recognition. In ICASSP Signal Processing (ICASSP), pp. 7054-7058. IEEE, 2020.
IEEE International Conference on Acoustics, Speech and
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint
2018
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ast corpus based public domain audio books. In IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 5206-5210, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Chemg Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: /904.0:00477077 08779, 2019.
Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding. In International Conference Machine Learning, pp. 17627-17643. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
Heli Qi, Sashi Novitasari, Sakriani Sakti, and Satoshi Nakamura. Improved consistency training for semi-supervised srruuecce-o--equenee asr via speech chain reconstruction and transcransc...... arXiv preprint wxXiv:205.006666 2022.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016.
Haşım Sak, Andrew Senior, Kanishka Rao, Ozan Irsoy, Alex Graves, Françoise Beaufays, and Jo- han Schalkwyk. Learning acoustic frame labeling for speech recognition with recurrent neural networks. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4280-4284, 2015.
Ashtosh Sapru. Using data augmentation and consistency regularization to improve semi-supervised speech recognition. 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pp. 1715-1725, 2016.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929 1958, 2014.
Ryoichi Takashima, Li Sheng, and Hisashi Kawai. Investigation of sequence-level knowledge dis- tillation methods for etc acoustic models. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Pp. 6156-6166 6160. IEEE, 2019.
Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski, Jakob Verbeek, and Hervé Jégou. Co-training submodels for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11701-11710, 2023.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 1 1240-1224- 2011
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of Inter- speech, pp. 2207-2211, 2018.
Felix Weninger, Franco Mana, Roberto Gemello, Jesús Andrés-Ferrer, and Puming Zhan. Semi- supervised learning with data augmentation for end-to-end ast. arXiv preprint arXiv:2007.13877 2020.
Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. d-drop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems, 34:10890-10905, 2021.
Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang. Zengrui Jin, Long Lin, and Daniel Povey. Zipformer: A faster and better encoder for automatic speech recognition. In The Twelfth International Conference on Learning Representationss 2024.
Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. WeNet: Production Oriented Streaming and Non-Streaming End-to- End Speech Recognition Toolkit. In Proc. Interspeech, pp. 4054-4088 2021.
Won Yoon, Hyeonseung Lee, Ju Yeon Kang, and Nam Soo Kim. Cons-kd: Dropout-robust knowledge distillation for ete-based automatic speech recognition. IEEE Access, 2024.
14

---

Published as a conference paper at ICLR 2025
Piotr Zelasko, Daniel Povey, Jan Trmal, Sanjeev Khudanpur, et al. Lhotse: speech data represen- tation library for the modern deep learning ecosystem. arXiv preprint arXiv:2110.12561, 2021.
Binbin Zhang, Di Wu, Zhendong Peng, Xingchen Song, Zhuoyuan Yao, Hang Lv, Lei Xie, Chao Yang. Fuping Pan, and Jianwei Niu. Wenet 2.0: More productive end-to-end speech recognition toolkit. arXiv preprint arXiv:2203.15555, 2022.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVE international conference computer vision, pp. 3713-3722, 2019.
Xiatian Zhu, Shaogang Gong, al. Knowledge distillation by on-the-fly native ensemble. Advances in neural information processing systems, 31, 2018.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
SMOOTI--REGULARIZDD CTC
CTC (SR-CTC) discourages peaky distributions by adding an smooth regular- ization loss (denoted as Csn) to regular CTC model. Specifically, we first apply smooth kernel K of size 3 to the model prediction 2, smoothing it along the time dimension: 2!*) smooth K).
The smoothing operation is done by using a 1-D depth-wise convolution layer. Then we minimize the DKL. between and 2s²), similar the consistency loss in CR-CTC (Equation 4):
T
(z x()))= Σ (sg(()(( (()
(5)
The overall loss of SR-CTC is formulated as:
L' = Lcro(Zzy) + BL C.sn(2.. x())),
(6)
where | is
In this work, we use K = (0.25, 0.5, 0.25) and / = 0.2.
We validate its effectiveness in Section
Table 6 presents the experimental result.
A.2 TRAINING CONFIGURATION
Training configuration, including the number of GPUs and training epochs, on LibriSpeech, Aishell- and GigaSpeech datasets are presented in Table Table
and Table 10 respectively.
Table 8: Training configuration on LibriSpeech dataset.
GPUs (80G NVIDIA Tesla A100)
Model
Epochs
CTC, Zipformer-S
100
CTC, Zipformer-M
100
CTC, Zipformer-L.
2
100
CR-CTC, Zipformer-S
50
CR-CTC, Zipformer-M
2
50
CR-CTC, Zipformer-L
50
CR-CTC/AED, Zipformer-L.
50
Pruned transduccr w/ CR-CTC, Zipformer-L.
2
50
Table 9: Training configuration on Aishell-1 dataset.
GPUs (80G NVIDIA Tesla A100)
Model
Epochs
CTC, Zipformer-S
120
CTC, Zipformer-M
120
CTC/AED. Zipformer-S
60
CTC/AED, 7ipformer-M
60
CR-CTC, Zipformer-S
60
CR-CTC, Zipformer-M
60
A.3 RESULTS OF DIFFERENT DECODING METHODS
Results comparison between greedy search decoding and prefix search decoding for CTC and CR- CTC on LibriSpeech, Aishell-1 and GigaSpeech datasets presented in Table [T] Table 12, and Table 13 respectively. addition, Table [4 presents the results of different beam sizes for prefix search decoding LibriSpeech dataset with Zipformer-M encoder.
16

---

Published as a conference paper at ICLR 2025
Table 10: Training configuration on GigaSpeech dataset.
GPUs
Model
Epochs
(80G NVIDIA Tesla A100)
CTC, Zipformer-S
2
60
CTC. Zipformer-M
2
60
CTC. Zipformer-L
2
60
CTC, Zipformer-XI.
60
CTC/AED, Zipformer-S
2
30
CTC/AED, Zipformer-M
2
30
CTC/AED, Zipformer-L
30
CTC/AED, Zipformer-XI.
30
Pruned transducer, Zipformer-S
2
30
Pruned transducer, Zipformer-M
2
30
Pruned transducer, Zipformer-I
30
Pruned transducer, Zipformer- XL
4
30
CR-CTC, Zipformer-S
2
30
CR-CTC, Zipformer-M
30
CR-CTC, Zipformer-L
2
30
CR-CTC, Zipformer-XL
4
30
CR- CR-CTC/AED, Zipformer
30
Pruned transducer w/ CR-CTC, Zipformer-X XL
30
Table 11: WER (%) results of different decoding methods on LibriSpeech dataset.
Greedy search decoding test-clean
Prefix search decoding
Model
test-other
test-clean
test-other
CTC, Zipformer-S
6.91
2.85
6.89
CTC, Zipformer-M
2.51
6.02
2.52
6.02
CTC, Zipformer-L
2.49
5.7
2.5
5.72
CR- TR-CTC, Zipformer-S
2.57
5.95
2.52
5.85
CR CR-CTC, Zipformer-M
2.12
4.62
2.1
4.61
CR-CTC, Zipformer-L. 2.03
4.37
2.02
4.35
Table 12: WER (%) results of different decoding methods on Aishell-1 dataset.
Greedy scarch decoding dev
Prefix search decoding
Model
test
dev
lest
CTC, Zipformer-S
4.88
5.26
4.89
5.26
CTC, Zipformer-M
4.46
4.8
4.47
4.8
CR-CTC, Zipformer-S
3.9
4.12
3.9
4.12
CR-CTC, ZipformerMM
3.73
4.02
3.72
4.02
Table 13: WER (%) results of different decoding methods GigaSpeech dataset.
Greedy search decoding dev
Prefix search decoding
Model
test
dev
test
CTC, Zipformer-S
12.15
12.03
12.08
11.95
CTC, Zipformer-M
11.3
11.31
11.23
11.27
CTC, Zipformer-L
11.21
11.19
11.16
11.16
CTC, Zipformer-XI.
10.85
10.91
10.8
10.87
CR-CTe CTC, Zipformer-S
11.85
11.8
11.68
11.58
CR-CTC, Zipformer-M
10.78
10.88
10.62
10.72
CR-CTC, Zipformer-L.
10.42
10.56
10.31
10.41
CR-CTC Zipformer-XL XL
10.28
10.41
10.15
10.28
17

---

Published as a conference paper at ICLR 2025
Table 14: WER (%) results of different beam sizes for prefix search decoding on LibriSpeech dataset using Zipformer-M encoder.
Method
Beam size
test-clean
test-other
2.52
6.02
CTC
2
2.52
6.02
2.52
6.02
8
2.52
6.02
2.1
4.61
CR-CTC
2.1
4.61
4
2.1
4,61
2.1
4.61
A.4
MODEL CONFIGURATION OF DIFFERENT SCALES OF ZIPFORMER
Table 15
presents model configuration of different scales of Zipformer.
Table 15: Model configuration of Zipformer four different scales.
Scale
layer-numbers
dineedonng-dineesioons
fead-forwarddddimensioon
(2,2,2,2,2.2)
11922566.266226,25622666
7512.768.768,768,766 768,768)
M
(2,2,3,433.2]
192.256.384.5122.88444
(512,768, 1024, 153,, 1024,768)
(2,2,4,5,422 (2,2,4,54422)
(1922256.5122,68,51222355
(512,768, 1036,2088, 1536,768
XI.
1992,84,,68,,124,,688888
1512,1002,2088,37222222 1024]
A.5 ABLATION STUDIES ON HYPER-PARAMETER TUNING
Table 16 presents results of tuning hyper-parametens, including in Equation and the ratio used to increase the amount of time masking for CR-CTC.
Results of tuning that controls LCR (Equation 3) and the ratio used to increase the amount of maseina for CR-CTC on LibriSpeech dataset using Zipformer-M encoder and greedy search decoding.
Table 16:
WER (%)
Hyper-parameter
test-clean test-other
Q 0.1
2.19
4.8
a 0.2 (final)
2.12
4.62
Q 0.3
2.23
4.84
1.0x time masking
2.19
4.98
1.5x time masking
2.19
4.73
2.0x time masking
2.17
4.71
2,5x time masking (final)
2.12
4.62
3.0x time masking
2.17
4.81
A.6 EMA-DISTILLED CTC
In EMA-distilled CTC, the teacher model ((r) is dynamically constructed for self-distillation. Its weights g(c) are updated using the exponential moving average of the current model's weights 0: 10/ max(20, step)). The teacher model f(()) processes the unmasked input (x) and produces the CTC distribution ((*) = (x(()) which distillation target for the current model f. Similar to Len in CR-CTC (Equation 4) the
g(e) < Tg(e) + (1 - T)0, where = min(0. (0.9999
18

---

Published as a conference paper at ICLR 2025
distillation loss CEMA is defined as:
LEMA (z, (0))
Σ
DEL DKL (sg(=t [11=()
1=1
The overall loss of EMA-distilled CTC is formulated as:
L" = C-rcccc + 7/EMA (t)),
(8)
where y hyper-parameter In this work, we use 7 0.2. Table 4 presents the experimental result.
A.7 RESULTS USING CONFORMER ENCODER
To validate the effectiveness and generalization ability of our proposed CR-CTC, we conduct ex- periments on LibriSpeech dataset using Conformer (Gulati al. 2020) encoder, comparing dif- ferent methods including CTC (Graves et al. (006), CTC/AED (Watanabe et al. 2017), pruned transducer (Kuang al. 2022) and CR-CTC. Specifically, we use a 12-layer Conformer, with an embedding dimension of 512, convolution kernel size of 31, and feedforward hidden dimension of 2048. For the CTC/AED model, the AED decoder is modeled by a 6-layer Transformer, where each layer has an attention dimension of 512 and feedforward hidden dimension of 2048. The vanilla CTC model trained for 100 epochs, while the other three models are trained for 50 epochs. Table 17 presents the experimental results. CR-CTC substantially outperforms the vanilla CTC and achieves marginally better results compared to the pruned transducer and CTC/AED.
Table 17: WER(%) performance of difference methods on LibriSpeech dataset using a 12-layer Conformer encoder.
WER (%)
Method
Params (M)
test-clean
test-other
CTC
77.4
2.92
7.15
CTC/AED
103.1
2.5
5.94
Pruned transducer
78.6
2.49
5.87
CR-CTC (ours)
77.4
2.43
5.78
19


Reviews:
[{"review_id": "mWNjKAYuiO", "reviewer": "Reviewer_LMc2", "summary": "This paper applies a new type of self-consistency loss on different augmented view for CTC based ASR model. The new consistency regularized loss is doing KL over CTC output. Experimental results shows WER got improved.", "questions": "Please properly cite literature work and make proper comparison.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed idea is intuitive. Although the gain is small, the paper is compared with couple competitive baseline on librispeech.", "weaknesses": "The paper lack of citation to many very relevant work: \n\nMost important one:\nContrastive siamese network for semi-supervised speech recognition (https://arxiv.org/pdf/2205.14054). The paper focus on compare with SoTA, instead of compare with literature self-distill or other consistency based baseline. In that paper, it include practical trick to make SimSiam type of model work for ASR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iWzJP3GVvH", "reviewer": "Reviewer_6WyR", "summary": "This paper proposes an improved version of Connectionist Temporal Classification (CTC) called Consistency-Regularized CTC (CR-CTC) for automatic speech recognition (ASR). CR-CTC enforces consistency between two CTC distributions obtained from different augmented views of the input speech mel-spectrogram.  The proposed method has 3 advantages: 1) it conducts self-distillation between random pairs of sub-models that process different augmented views; 2) it learns contextual representation through masked prediction for positions within time-masked regions, especially when we increase the amount of time masking; 3) it suppresses the extremely peaky CTC distributions, thereby reducing overfitting and improving the generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of CR-CTC, which achieves performance comparable to, or even slightly better than, that of transducer and CTC/AED.", "questions": "For the combination of transducer and CR-CTC model, does the CTC score is used during the decoding?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "CR-CTC takes two different augmented views of the same speech mel-spectrogram as the inputs and enforce consistency between the two obtained CTC distributions. This method helps the model to do self-distillation between randomly sampled sub-models, learn contextual representation through masked prediction and reduce the peaky CTC distribution. The idea is simple and easy to implement. The training cost didn’t increase based on the description in section 4.1. The experiments on LibriSpeech, Aishell-1, and GigaSpeech show the proposed method outperform standard CTC in all these sets for different model sizes. It achieves comparable accuracy or even better than the advanced transducer or CTC/AED model. The paper also provided detailed ablation study to help the reader understand more details about the method.", "weaknesses": "Comparing results in table 1 and 3, the advantages of CR-CTC over standard CTC is smaller for GigaSpeech set than that for LibriSpeech set.  This may indicate that the proposed method may not work very well for big training data, e.g. tens of thousands of speech hours. \n    In table 1, 2 and 3, it’s not mentioned that the results for transducer and CTC/AED models are from beam search or greedy search. For these two models, the results of beam or greedy search usually have big differences. If the results given are from greedy search, it may mean the accuracy of CR-CTC model still have gap from that of transducer and CTC/AED model with beam search.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dRD5GZir0p", "reviewer": "Reviewer_HiGE", "summary": "Consistency regularization (CR) is a well established existing method, where you forward through some model two times with different augmentation (and maybe also dropout or other randomness) to get two predictions, and then you minimize the symmetric KL between both, or similar. Thus, this method is purely on the training side, and doesn't change any modeling aspect.\n\nHere, CR is applied to speech recognition, specifically to CTC models, on a frame-by-frame basis, called CR-CTC. The main difference in the two branches is caused by different SpecAugment masking.\n\nFor fair comparison, due to forwarding the data twice now, the number of epochs and the batch size are both halfed for the CR-CTC case.\n\nExperiments are done on Librispeech, Gigaspeech and Aishell.\n\nThe method is mainly tested on CTC, but then some extension to that is when they used a joint AED/CTC model in the end, where CR is applied only to the CTC part.\n\nA number of ablations has been made on the loss scale and on increasing the SpecAugment masking, which seems ot help more on CR-CTC, but for pure CTC, the original amount of SpecAugment masking already seems optimal.\n\nThe improvements on Librispeech test-other are quite large: From 5.72% WER to 4.35% WER.\n\nIt is also shown that it reduces the peakiness of the alignment behavior a bit.", "questions": "Abstract starts a bit strange. It says CTC is worse than RNN-T and AED. Yes sure, we know. But then it talks about some method to maybe improve CTC. So why is mentioning RNN-T/AED relevant? Is it because you think the gap between CTC and AED/RNN-T is larger than what you would expect, and some methods like the presented here should close the gap? But I don't think that this really is being shown here in this work. Also, some variant of this method could maybe be applied to AED/RNN-T just as well. So, I don't really see why mentioning AED/RNN-T in the abstract is really relevant for this work here. It's fine in the introduction, to put CTC into perspective, but I don't think it's relevant in the abstract. I was a bit confused about this.\n\nEq 3 and also Figure 1, the CTC loss is maybe better formulated on z, not on x? I found it weird that x goes into L_CTC but z goes into L_CR.\n\n\n\"a time warping factor of 80\" - what does that mean? I don't think you make the sequence 80 times longer?\n\nPlease clarify the downsampling of the Zipformer. Do you stick to the original Zipformer here, where the Conv Frontend downsamples the 100Hz feature frames to 50Hz, and then the residual/bypass connection is always at 50Hz, and at the very end, you downsample again to get 25Hz output frames, i.e. the log probs are at 25 Hz?\n\nDid you investigate how the downsampling influences the CR-CTC loss? I think this can have quite a crucial impact, as we know that in general, for CTC, the ratio of input length to output length plays an important role for convergence and training dynamics.\n\nDid you also test with other vocab sizes? 500 BPE size seems quite small to me. How does it perform with larger vocab sizes, e.g. with 10k?\n\nHow does the Zipformer influence the results? Specifically, do you think you get the same improvements with a normal Conformer?\n\n\n\"auxiliary head of AED\" (p9, l483) (and also same with transducer) / Table 7: I don't exactly understand what you report here. Is the AED (or transducer) head just used as an aux loss, and during recognition, you only use the CTC head and ignore the AED (or transducer)? Please be more clear about that. Also, you are giving the wrong citation for that. The reference you give is about joint AED/CTC, where both AED and CTC heads are used for recognition, so nothing is ignored, nothing is just used as aux loss. The only reference I know where AED is used as an aux loss for CTC is \"Keep Decoding Parallel With Effective Knowledge Distillation From Language Models To End-To-End Speech Recognisers\", Hentschel et al, 2024.\n\n\nOn GigaSpeech, improvement seems much less (XL, test: 10.87 -> 10.28) compared to Librispeech (5.72 -> 4.35). Why?\n\nTable 3 caption: \"GigaSpeeech\" typo.\n\n\nTransducer w/ CR-CTC, what exactly is that? The same approach applied on transducer? But then this is not CTC? Or is it combined CTC with transducer?\n\n\nNote, as your method is very simple to implement, and your improvements here are really impressive, I was just trying it out myself. However, with negative result: For my Conformer CTC baseline, on 100Hz inputs, downsampled by factor 6, with BPE 10k vocab, with aux AED loss (\"Keep Decoding Parallel With Effective Knowledge Distillation From Language Models To End-To-End Speech Recognisers\", Hentschel et al, 2024), where my baseline with greedy decoding without LM was at 5.93% on dev-other, it degraded with CR-CTC to 5.99% on dev-other. I halved the number epochs and halved the batch size for the CR experiment, just like you did. This is with CR loss scale 0.2. I did not adapt SpecAugment yet, but from your paper, I would expect that even with this setting, I should already see quite some improvement. So, why don't I? Your paper is lacking such study on other settings, as mentioned above (Conformer, other BPE sizes, other downsampling) to know whether I can/should expect similar improvements there or not, and whether I maybe need a very different CR loss scale there, or whether I need to care about other things.\n\n\nNote, halving the batch size can have other effects. Many methods (e.g. optimizer, LR schedule, regularization, etc) don't work in the same way for different batch sizes. You do effectively more updates to the model. It can also have a regularization effect. So, I think an important missing experiment is: What happens to the baseline when you half the batch size? Maybe you also get improvements there?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple method.\n\n- Seems to give huge improvements, at least in some settings.", "weaknesses": "- Some smaller details are a bit unclear.\n- Only tested for Zipformer. More standard would be Conformer, but this is missing.\n- Unclear how well this method works in other cases, e.g. other models, other datasets, some other hyperparams different. Specifically, I tested it in my setup, and it didn't really helped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CqQAgod9AX", "reviewer": "Reviewer_LPhF", "summary": "This paper proposes a method to improve CTC performance by applying self-distillation between sub-models using drop-based techniques. The approach aims to enhance target token distribution predictions within time-masked regions and develop contextual representations from unmasked segments, drawing inspiration from self-supervised learning methods. By increasing time-masking, this technique promotes effective masked prediction, reducing peaky CTC distributions and strengthening the model's generalization ability. Experiments on multiple datasets—Librispeech, Aishell-I, and GigaSpeech—demonstrate that the proposed method achieves performance on par with transducer and CTC/AED models when used in joint training.", "questions": "* Why was the choice of alpha set to 0.2 in Equation 3? It would benefit readers if the authors could provide results from an ablation study showing the impact of different alpha values on performance. This would offer greater insight into the method's sensitivity to this hyperparameter choice.\n* Why was a beam size of 4 specifically chosen for comparisons with other state-of-the-art models? The authors may consider including results with different beam sizes (e.g., 1, 4, 8) in an appendix to show the method's sensitivity to this parameter.\n* The authors employed a larger amount of time-masking by increasing both the number of time-masking regions and the maximum masking fraction by a factor of 2.5. However, it would be interesting to know how much time-masking is optimal for masked prediction. The authors could provide results from an ablation study showing performance with different amounts of time-masking (e.g., 1x, 1.5x, 2x, 2.5x, 3x) for ZipformerXL and at least one baseline model in Table 15. This would help readers understand how critical this choice is and how generalizable the method is.\n* In Tables 1, 2, and 3, are the CTC/AED baseline results reported with CTC-only decoding or CTC/AED joint decoding? This could be clarified by specifying the decoding methods for the baselines and the proposed method in the table captions.\n* For a deeper understanding, the authors could include results showing the impact of increased time-masking on the baseline CTC model as well. This would help isolate whether the benefit comes from the two-branch architecture of CR-CTC or simply from more aggressive augmentation. Although the authors reported one of the baselines with larger time-masking, it would be helpful if results were provided for the other tables as well.\n* In Table 3, the best results were obtained using Zipformer XL. However, the authors should:\n1. Explain the rationale for using Zipformer-M in Tables 4 and 5 instead of Zipformer-XL.\n2. Provide results for Zipformer-XL in Table 11 for completeness.\n3. Clarify whether the results in Table 11 are from self-distillation or masked prediction in CR-CTC.\n* The results of SR-CTC in Table 6 are slightly worse than those of CR-CTC. Do the authors have any explanation for this behavior, and does SR-CTC also use increased time-masking?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper presents a simple yet effective distillation method by extending time-masking to develop contextual representations from unmasked segments across two different augmented views.\n* It demonstrates that CTC performance is comparable to that of transducer and CTC/AED models.", "weaknesses": "The method generates two different augmented views by independently applying existing SpecAugment to Zipformer. However, it raises the question of how generalizable this claim is when applied to other architectures like Conformer, E-Branchformer, or Branchformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)