OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024)

Abstract: There has been an increasing interest in large speech models that can perform multiple tasks in a single model. Such models usually adopt an encoder-decoder or decoder-only architecture due to their popularity and good performance in many domains. However, autoregressive models can be slower during inference compared to non-autoregressive models and also have potential risks of hallucination. Though prior studies observed promising results of non-autoregressive models for certain tasks at small scales, it remains unclear if they can be scaled to speech-to-text generation in diverse languages and tasks. Inspired by the Open Whisper-style Speech Model (OWSM) project, we propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC). It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID). Compared to encoder-decoder OWSM, our OWSMCTC achieves competitive results on ASR and up to 24% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference. OWSM-CTC also improves the longform ASR result with 20x speed-up. We will publicly release our code, pre-trained model, and training logs to promote open science in speech foundation models.1

heading: 1 Introduction
text: The great success of large language models (LLMs) (OpenAI, 2023; Touvron et al., 2023; Anil et al., 2023b) has sparked a growing interest in developing foundation models in various modalities. Recent studies have explored different approaches towards multilingual and multi-tasking speech foundation models (Radford et al., 2023; Zhang et al., 2023; Pratap et al., 2023; Rubenstein et al., 2023; Barrault et al., 2023; Peng et al.,
1https://github.com/espnet/espnet
2023e). OpenAI Whisper (Radford et al., 2023) is a series of Transformer encoder-decoder models trained on 680k hours of proprietary labeled audio. Whisper achieves strong results in multilingual automatic speech recognition (ASR), any-toEnglish speech translation (ST), and spoken language identification (LID). Although it shows the effectiveness of large-scale (weakly) supervised pre-training, the full development pipeline, including training data details, is not publicly accessible. Recent works have developed Open Whisperstyle Speech Models (OWSM) (Peng et al., 2023e, 2024) with the aim of reproducing Whisper-style training using public data and open-source toolkits. However, Whisper and OWSM adopt the encoderdecoder architecture, which generates text tokens given speech in an autoregressive manner. They might hallucinate during inference, and the speed
10192
can be slow. Other models with decoder-only architectures, like AudioPaLM (Rubenstein et al., 2023) and VioLA (Wang et al., 2023b), could suffer from the same issues due to autoregressive decoding.
Another type of work like Google USM (Zhang et al., 2023) and Meta MMS (Pratap et al., 2023) uses non-autoregressive models with Connectionist Temporal Classification (CTC) (Graves et al., 2006), but these CTC-based models are designed for ASR only. Prior studies have also achieved promising results of CTC models for ST only, but they mainly focus on specific language pairs at much smaller scales (Inaguma et al., 2021; Chuang et al., 2021; Xu et al., 2023). Some of them employ additional decoders (Inaguma et al., 2021; Yan et al., 2023) or cross-attention layers (Xu et al., 2023), making the model more complicated.
A natural question now arises: Can we build a non-autoregressive encoder-only model for speechto-text generation in diverse languages and multiple tasks like Whisper/OWSM? This research problem has become increasingly important in the era of LLMs because large-scale pre-trained speech encoders can serve as an adapter between the speech and text modalities (Gong et al., 2023; Wang et al., 2023a), providing a promising avenue towards general-purpose multi-modal foundation models (Anil et al., 2023a).
In this work, we propose OWSM-CTC, a novel encoder-only speech foundation model based on multi-task self-conditioned CTC to imitate OWSM’s multilingual ASR, any-to-any ST, and LID functionalities. Following previous encoderdecoder OWSM v3.1 models (Peng et al., 2024), we train a 1B OWSM-CTC model using 180k hours of public data covering 151 languages. Extensive evaluations show that our OWSM-CTC exhibits strong performance and efficiency. Compared to the 1B OWSM v3.1 medium model, OWSM-CTC achieves comparable performance for ASR and superior performance for various ST directions (up to 24% relative improvement) while being more robust and showing 3 to 4 times inference speedup. OWSM-CTC also improves the WER for longform ASR and can be 20 times faster due to batched parallel decoding. OWSM-CTC further outperforms the other baseline models on LID. Our code, pre-trained model weights, and training logs will be publicly released to facilitate the development of large speech models.

heading: 2 Related Work

heading: 2.1 Speech foundation models
text: Attention-based encoder-decoder. OpenAI Whisper (Radford et al., 2023) adopts the standard Transformer encoder-decoder architecture (Vaswani et al., 2017) and scales the training data to 680k hours of proprietary labeled audio.2 However, the complete pipeline for model development, including training data details and training code, is not publicly available. A recent project, OWSM, aims to reproduce Whisper-style training using public data and open-source toolkits to promote transparency and open science in this field (Peng et al., 2023e). The latest OWSM v3.1 models (Peng et al., 2024) employ E-Branchformer (Kim et al., 2023) as the encoder and Transformer as the decoder, which are trained with a joint ASR CTC loss (Kim et al., 2017). Although OWSM has promising results using public corpora, it still follows the encoder-decoder architecture, which can be slow and unstable at inference time. Decoder-only. Several studies employ decoderonly models for speech-to-text tasks. AudioPaLM (Rubenstein et al., 2023) extends the textual PaLM-2 (Anil et al., 2023b) to support speech understanding and generation tasks including ASR and ST. DOTA (Gupta et al., 2024) is a decoderonly Transformer model trained on 93k hours of public English ASR data, but it does not support other languages or ST. Decoder-only models face the same slowness and robustness issues as encoderdecoder due to autoregressive decoding. CTC or Transducer. Another line of research proposes to utilize CTC (Graves et al., 2006) or Transducer (Graves, 2012) for ASR. Google USM (Zhang et al., 2023) provides generic ASR models that are first pre-trained on 12M hours of unlabeled audio and then fine-tuned on proprietary labeled data with CTC or Transducer. Meta MMS (Pratap et al., 2023) pre-trains a wav2vec 2.0 model (Baevski et al., 2020) on massively multilingual data and then fine-tunes it with CTC on labeled ASR data covering over 1k languages. These models employ CTC only for ASR. In our OWSMCTC, we propose a single CTC-based encoder-only model for ASR, ST, and LID. Our supported tasks are more similar to Whisper-style models.
2Their latest large-v3 version uses 1M hours of labeled audio and 4M hours of pseudo-labeled audio.

heading: 2.2 Efficient speech models
text: Model compression. Various algorithms have been utilized to compress speech models, including knowledge distillation (Chang et al., 2022; Lee et al., 2022; Peng et al., 2023d; Gandhi et al., 2023), pruning (Lai et al., 2021; Peng et al., 2023a), quantization (Yeh et al., 2023; Ding et al., 2023), and dynamic module execution (Yoon et al., 2022; Peng et al., 2023c; Strimel et al., 2023). These methods are typically applied to pre-trained models and are thus orthogonal to this work. In the future, we will apply compression to further improve efficiency. Efficient architectures. Better network architectures can also improve efficiency, including attention with linear complexity (Beltagy et al., 2020; Wang et al., 2020b; Tay et al., 2023) and sequence length reduction (Burchi and Vielzeuf, 2021; Kim et al., 2022; Nawrot et al., 2023; Rekesh et al., 2023). In this work, we do not modify the attention but use larger downsampling in the convolution module to reduce the sequence length. More details are in Appendix A.2 and B.1.

heading: 2.3 CTC-based speech models
text: Non-autoregressive models have a faster inference speed than their autoregressive counterparts due to parallel decoding. They have been utilized in machine translation (Gu et al., 2018; Ghazvininejad et al., 2019; Xiao et al., 2023), ASR (Chen et al., 2019; Higuchi et al., 2020; Ng et al., 2021; Chi et al., 2021; Lee and Watanabe, 2021; Nozaki and Komatsu, 2021), and ST (Inaguma et al., 2021; Chuang et al., 2021; Xu et al., 2023).
CTC is originally proposed to label sequences without explicit segmentation (Graves et al., 2006). CTC-based ASR models learn a monotonic alignment between speech features and text tokens. With parallel greedy decoding, they are much faster than autoregressive models. However, the accuracy of CTC is generally inferior due to the conditional independence assumption between output tokens. To address this issue, Intermediate CTC (InterCTC) (Lee and Watanabe, 2021) calculates additional CTC losses using intermediate representations from the encoder. Self-conditioned CTC (Nozaki and Komatsu, 2021) further extends InterCTC by adding back predictions of intermediate CTC layers to the subsequent encoder. These approaches have shown to be highly effective in speech-to-text generation tasks without a decoder (Higuchi et al., 2021).
Although CTC assumes a monotonic alignment between input and output, it can be used for ST with the reordering capability of self-attention (Inaguma et al., 2021; Chuang et al., 2021).
Conventional CTC models are typically designed for a specific task or language. It remains under-explored whether such approaches can be scaled to multilingual and multi-task scenarios. This work proposes a novel encoder-only speech foundation model based on multi-task selfconditioned CTC. This single model performs well in multilingual ASR, ST, and LID.

heading: 3 OWSM-CTC

heading: 3.1 Overall architecture
text: Figure 2 shows the architecture of OWSM-CTC. Its main component is a speech encoder, which takes speech features as input and predicts the spoken language as well as the ASR or ST hypothesis using CTC. To mimic Whisper-style models that condition text generation on an optional text prompt (Radford et al., 2023; Peng et al., 2023e, 2024), we employ a separate Transformer encoder to process the prompt and inject the output to the main model through cross-attention. Then, the model can potentially attend to the text prompt when generating text.

heading: 3.2 Speech encoder
text: For an input waveform, we first extract log Mel filterbanks and then apply a 2D convolution module to downsample the feature sequence along the time dimension. Let Xspeech ∈ RT×d be the downsampled feature sequence of length T and feature size d. To specify the language and task, we prepend two special tokens to the sequence:
X = concat(elang, etask,Xspeech), (1)
where concat(·) is concatenation along time and elang, etask ∈ R1×d are embeddings of special tokens <lang> and <task>, respectively. X now has shape (T + 2) × d. If the spoken language is known, the true language token will be used as input. Otherwise, a special token <nolang> denoting “unknown language” will be used. During training, we randomly replace the true language with <nolang> according to probability 0.5 so that either can be used for inference. The task token is <asr> for speech recognition and <st_lang> for translation to a target language.
Next, we add sinusoidal positional embeddings to X, and apply a stack of N encoder layers:
X(0) = X+ PosEmb(X), (2)
X(l) = SpeechEnc(l)(X(l−1)), (3)
where l is a layer index from 1 to N , PosEmb(·) generates positional embeddings, and SpeechEnc(l)(·) is the l-th encoder layer. The encoder is E-Branchformer (Kim et al., 2023), an enhanced version of Branchformer (Peng et al., 2022), which shows excellent performance across a wide range of benchmarks (Peng et al., 2023b).
We compute the CTC loss using the final encoder output X(N) and an augmented reference ytask. To create this reference, we simply preprend <lang> and <task> to the original groundtruth text of the desired task. Hence, the model will learn to predict the language token in addition to ASR or ST text tokens. This CTC loss is denoted as follows:
L(N) = − logPCTC(ytask | softmax(X(N)W1)), (4)
where W1 ∈ Rd×V is a linear layer and V is the size of the CTC vocabulary.
As discussed in Section 2.3, we apply selfconditioned CTC (Nozaki and Komatsu, 2021) at intermediate layers S ⊆ {1, . . . , N − 1} to alleviate the conditional independence assumption of CTC. For any layer s ∈ S, Equation 3 is replaced
by the following operations:
A(s) = SpeechEnc(s)(X(s−1)), (5)
B(s) = softmax(A(s)W1), (6)
X(s) = A(s) +B(s)W2, (7)
where W2 ∈ RV×d is a linear layer. The intermediate CTC loss at layer s is defined as follows:
L(s) = − logPCTC(y(s) | B(s)), (8)
where y(s) is the augmented reference at layer s. Similar to ytask in Equation 4, we prepend the language and task tokens to the original groundtruth text. Note that the choice of the reference text depends on the task. If the task for the current input is ASR, we simply use the ASR transcript to create y(s) for all s, which is consistent with conventional ASR models. However, if the task is ST, we empirically find that the model cannot converge if we use the translated text as the reference at all intermediate layers S (see Appendix B.2 for discussions). Therefore, as shown in Figure 2, we utilize the ASR transcript at the first NASR layers and the ST text at the remaining NST layers, where NASR +NST = |S| ≤ N − 1. This design mimics a cascaded system that first performs ASR and then ST, but our entire model is optimized jointly and trained from scratch. In other words, the first NASR CTC layers always perform ASR regardless of the task token (named “ASR-only CTC”), whereas the
other CTC layers are multi-tasking - they can perform ASR or ST according to the task token (named “task-specific or task-dependent CTC”).
The overall training loss is an average of the loss terms defined in Equation 4 and Equation 8:
Ltotal = 1
1 + |S|
( L(N) + ∑
s∈S L(s)
) . (9)

heading: 3.3 Prompt encoder
text: Whisper-style models generate text conditioned on an optional text prompt (Radford et al., 2023; Peng et al., 2023e, 2024). During training, this prompt is simply the previous sentence in the same audio recording. During inference, it can be provided by the user to potentially adjust the output. For encoder-decoder models like Whisper, the text prompt is a prefix to the autoregressive decoder. For our encoder-only model, we leverage a separate Transformer encoder to process the prompt and inject it to the speech encoder through cross-attention. If no prompt is provided, a special token <na> will be used. Let Xprompt ∈ RT ′×d′ be the output of the prompt encoder. We insert a cross-attention layer at a subset of layers T ⊆ {1, . . . , N} of the speech encoder. For any t ∈ T , the original SpeechEnc(t)(·) in Equation 3 or Equation 5 becomes SpeechEncCA(t)(·, ·):
D(t) = SpeechEnc(t)(X(t−1)), (10)
SpeechEncCA(t)(X(t−1),Xprompt) =
D(t) + CrossAtt(D(t),Xprompt,Xprompt), (11)
where CrossAtt(·, ·, ·) is a cross-attention layer with three arguments: query, key, and value.
Our training data is a mixture of public ASR and ST datasets. Some of them provide unsegmented long audio, but the others only release segmented short audio. At training time, if a sample does not have a previous sentence, we will use <na>. Otherwise, we use either <na> or the previous sentence as the prompt according to 0.5 probability. Section 4.6 shows that OWSM-CTC can leverage the prompt’s information when necessary.

heading: 4 Experiments

heading: 4.1 Experimental setups
text: Table 1 is a brief summary of model size, training data, and training cost. Data format. Our training data is prepared using scripts publicly released by OWSM v3.1 (Peng
et al., 2024). It is a mixture of more than 25 public ASR and ST corpora covering 151 languages and various translation directions. The total audio duration is 180k hours. To create long-form data, consecutive utterances from the same audio recording are concatenated to a duration of no more than 30 seconds. The input audio to the model is always padded to a fixed length of 30 seconds. Appendix A.1 and Table 11 present the training data statistics. The original Whisper-style data contains the start and end timestamps for each utterance. These timestamp tokens are predicted along with normal text tokens during the autoregressive decoding. In OWSM-CTC, we do not include any explicit timestamps since the time-aligned hypothesis can be obtained by forced alignment if desired. Model architecture. Our speech encoder is a 27- layer E-Branchformer with a hidden size of 1024 and 16 attention heads. Four intermediate layers (6, 12, 15, and 21) are used for self-conditioned CTC. The first three are ASR only, while the others are task-specific. The prompt encoder is a 4- layer Transformer with a hidden size of 512 and 8 attention heads. It is injected into the speech encoder at every third layer. The total model size is 1.01B, which matches the size of the encoderdecoder OWSM v3.1 medium (1.02B). More details about the architecture are in Appendix A.2 (see Table 12). Implementation. We implement OWSM-CTC in ESPnet (Watanabe et al., 2018) based on PyTorch (Paszke et al., 2019). FlashAttention (Dao et al., 2022) is used to improve training efficiency, but it is not used for inference. The batch size per GPU is 4, and 64 NVIDIA A100 GPUs (40GB) are used with distributed data parallel. The total training time is approximately 300 hours. For optimization, we employ the Adam optimizer (Kingma and Ba, 2015) with the piece-wise linear learning
rate schedule (Peng et al., 2024). The peak learning rate is 2e-4. Other training hyperparameters can be found in Appendix A.3 (see Table 13). Evaluation. We fairly compare our encoder-only OWSM-CTC with the previously released encoderdecoder OWSM v3.1 models (Peng et al., 2024) since they are trained on the same data. We also show the results of Whisper under the same decoding setup for reference, but we note that they are not comparable with ours due to completely different training data. By default, short-form audio without any text prompt is used, but we also evaluate the long-form ASR performance in Section 4.5 and investigate the effect of text prompts in Section 4.6.

heading: 4.2 Language identification
text: Table 2 presents the LID results on the FLEURS test set (Conneau et al., 2023). Our OWSM-CTC achieves a top-1 accuracy of 87.6%, outperforming the other encoder-decoder models by a large margin. This is likely because spoken LID requires a powerful encoder to extract useful information from the input audio. Our encoder-only model is especially suitable for this type of task.

heading: 4.3 Speech recognition
text: Table 3 presents word error rates (WERs) on nine English ASR test sets. Following Peng et al. (2023e, 2024), we leverage greedy decoding and apply the Whisper English text normalizer before scoring.3 We record the average decoding time across all English test sets on an NVIDIA A40 GPU and calculate the relative speed-up. Results show
3We also report the results of Whisper large-v2 and OWSM v3.1 medium with beam search in gray for reference, but they are not comparable with the others due to different model sizes or decoding configurations. This applies to other tables as well.
that our non-autoregressive OWSM-CTC generally has comparable WERs with the autoregressive OWSM v3.1 medium (average: 7.9 vs. 7.7), both of which have 1B parameters. However, OWSM-CTC
achieves 3.63x speed-up due to parallel decoding. Notably, OWSM-CTC is even faster than OWSM v3.1 base, which has only 100M parameters, and our WERs are much lower (average: 7.9 vs. 12.1). Compared to Whisper models trained on significantly more data, our OWSM-CTC is still competitive in many cases, and our inference is much faster. These results demonstrate that OWSM-CTC achieves an excellent trade-off between recognition accuracy and inference efficiency.
Table 4 shows the results of multilingual ASR. We perform greedy decoding and apply the Whisper basic text normalizer before scoring. Our OWSM-CTC is slightly worse than OWSM v3.1 in terms of the average WER/CER (16.2 vs. 15.2). For European languages in MLS (Pratap et al., 2020), OWSM-CTC generally falls behind. But for East Asian languages like Chinese, Japanese, and Korean, OWSM-CTC is on par with or better than OWSM v3.1 medium. This difference might be related to the training data size and tokenization.

heading: 4.4 Speech translation
text: We evaluate ST on CoVoST-2 test sets (Wang et al., 2020a). By default, we perform greedy decoding and calculate BLEU scores in true case with punctuation.4 For X-to-En translation, we follow OWSM v3.1 (Peng et al., 2024) to report results of
4Results in lowercase without punctuation can be found in Appendix C, which are consistent with previous OWSM work (Peng et al., 2024).
directions where the training data size is over 100 hours. For the other low-resource directions, both OWSM v3.1 and our OWSM-CTC do not work in general. For En-to-X translation, we report all 15 directions. We calculate the speed-up based on the average decoding time on an NIVIDA A40 GPU.
Table 5 shows the X-to-En results. Notably, our encoder-only OWSM-CTC consistently outperforms the encoder-decoder OWSM v3.1 by a large margin. The average BLEU score is improved from 20.2 to 25.1 (24% relatively). We also achieve 3.35x speed-up for inference.
Table 6 presents En-to-X results. Whisper does not support these directions. Our OWSM-CTC achieves superior performance than OWSM v3.1 in 12 of 15 translation directions and most of them are statistically significant. The average BLEU is improved from 14.1 to 16.0 (13% relatively), and the inference speed-up is 4.20 times.
We have the following observations from the ST results: (1) Our non-autoregressive OWSM-CTC generally achieves 3 to 4 times speed-up compared to the encoder-decoder baseline, which is consistent with ASR. (2) OWSM-CTC even improves the ST performance sometimes by a large margin. One reason is that the autoregressive model suffers from hallucination and error propagation, while the non-autoregressive model is more stable. (3) The BLEU improvement of X-to-En is larger than that of En-to-X, likely because: (i) the OWSM training set contains lots of English ASR data and OWSMCTC might obtain strong capability of generating English text; (ii) X-to-En has fewer training data than En-to-X, and the encoder-decoder model may need a sufficient amount of training data to achieve good performance for translation.
Our findings reveal that large-scale CTC-based models are also promising for ST in various language pairs, which is consistent with prior investigations at smaller scales (Yan et al., 2023).

heading: 4.5 Long-form speech recognition
text: For long-form ASR, a model takes as input an unsegmented audio recording of arbitrary length and generates the entire transcription without explicit voice activity detection. Whisper and encoderdecoder OWSM can predict start and end timestamps of each utterance within a fixed-length segment. Those timestamps are used to shift the recognition window for chunk-wise long-form ASR. However, this chunk-wise recognition is a sequential process because the location of the next chunk
depends on the predicted timestamp in the current chunk.5 By contrast, our OWSM-CTC performs chunk-wise recognition in a fully parallel manner. We first split the entire audio into overlapped chunks of 30s, where the overlapped region serves as the left and right context.6 We then perform CTC greedy decoding on batched chunks. The batch size is 32 on a single NVIDIA A40 GPU (48GB). Table 7 shows the WER and speed-up with different context lengths. Our OWSM-CTC achieves lower WERs than the encoder-decoder OWSM v3.1, while being approximately 20 times faster due to the batched parallel decoding. OWSMCTC is also robust to different context lengths. These observations indicate that CTC-based non-
5The decoding process might be parallelized if tokenlevel timestamps are available. However, it remains an open problem to derive accurate token-level timestamps from an attention-based encoder-decoder model without extra training.
6We follow this tutorial for long-form ASR with CTC: https://github.com/NVIDIA/NeMo/blob/main/ tutorials/asr/Streaming_ASR.ipynb
autoregressive models perform very well for longform ASR, which is consistent with prior findings (Koluguri et al., 2023).

heading: 4.6 Effect of text prompt
text: As described in Figure 2 and Section 3.3, OWSMCTC can take an additional text prompt as input which might change the output. During training, either a special token <na> or the previous sentence in the same audio is used as the prompt according to a probability of 0.5, which follows the setup of Whisper and OWSM. To verify that OWSM-CTC can utilize information from the prompt when necessary, we perform greedy decoding on several test sets with the previous sentence in the dataset as a prompt. As shown in Table 8, using the previous sentence reduces the error rates. The p-values are computed using the Matched Pair Sentence Segment method.7 Appendix D provides an example where the previous sentence also affects the output text style.
7https://github.com/usnistgov/SCTK

heading: 4.7 Robustness
text: To investigate the robustness, we first consider random noise as input. Table 9 shows the ASR outputs generated by three models. Encoder-decoder models, including Whisper and OWSM v3.1, tend to generate some texts that look meaningful, while our OWSM-CTC generates fewer tokens, which are mostly punctuation marks that do not actually have meaning.
Another typical issue of autoregressive decoding is that the generation might fall into repetitions of a few characters or words. Table 19 in Appendix E presents two examples from ASR and ST, respectively. Our non-autoregressive model is more robust in such cases. To quantitatively measure this type of error, we consider a hypothesis as a failure if it contains any character-level θ-gram (θ = 1, 2, . . . , θmax) that consecutively occurs for at least δ times. Table 10 shows the number of failures in all ST test sets with different thresholds. We can see that the encoder-decoder OWSM v3.1 medium fails many times even with beam search, while our OWSM-CTC has almost no failures.

heading: 5 Conclusion
text: We propose OWSM-CTC, a novel encoder-only speech foundation model built upon 180k hours of public audio data and open-source toolkits. OWSM-CTC employs multi-task self-conditioned CTC for multilingual ASR, any-to-any ST, and LID. We conduct extensive experiments to compare OWSM-CTC with the encoder-decoder OWSM models trained on the same data. We find that OWSM-CTC achieves competitive performance on ASR and superior performance on ST for both X-to-En (24% relative improvement) and En-toX (13% relative improvement), while being more
robust and 3 to 4 times faster at inference time. Additionally, OWSM-CTC improves the long-form ASR WER with 20 times faster inference due to the batched parallel decoding. OWSM-CTC also outperforms the baselines on LID. To promote open research on large speech models, we will publicly release our code, pre-trained model weights and training logs.
Limitations
Although OWSM-CTC reduces the training cost by 22% compared to OWSM v3.1, it still requires nearly 20k GPU hours, which is nontrivial. OWSMCTC can generate incorrect ASR or ST outputs due to limited training data in certain languages. Care should be taken when using our model for low-resource ASR or ST. Besides, we have only evaluated our model with greedy decoding as it has the fastest inference speed. The non-autoregressive model sometimes makes mistakes in spelling or grammar due to a lack of language models.
Broader Impacts and Ethics
Our OWSM-CTC is a novel encoder-only speech foundation model built upon public datasets and open-source toolkits. Compared to other popular choices, it achieves very strong performance and efficiency. We adhere to the ACL ethics policy and there is no violation of privacy in our experiments. We plan to publicly release all scripts, pre-trained models, and training logs, which can promote transparency and open science. We believe this will benefit the entire speech research community and it can make the latest speech technology available to a broader range of people all over the world.

heading: Acknowledgements
text: Our computing resources are supported by PSC Bridges2 and NCSA Delta via ACCESS allocation CIS210014, under National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

heading: A Details of Experimental Setups
text: A.1 Training data
Table 11 summarizes the training data statistics. We prepare the training data mixture using the scripts publicly released by OWSM v3.1 (Peng et al., 2024). This ensures a fair comparison between our OWSM-CTC and the previously released encoder-decoder OWSM models.
Our use of the data is consistent with their intended use. These datasets have been widely used in speech research. They do not violate the privacy of creators or users, nor do they contain any
offensive content. Specifically, the individual training datasets and licenses are listed below: AIDATATANG (CC BY-NC-ND 4.0)8, AISHELL1 (Apache 2.0) (Bu et al., 2017), AMI (CC BY 4.0) (Carletta, 2007), Babel9, CommonVoice (CC01.0) (Ardila et al., 2020), CoVoST2 (CC BY-NC 4.0) (Wang et al., 2020a), Fisher Switchboard (LDC) (Godfrey et al., 1992), Fisher Callhome Spanish (LDC) (Post et al., 2013), FLEURS (CCBY-4.0) (Conneau et al., 2023), Googlei18n10, GigaSpeech (Apache 2.0) (Chen et al., 2021), GigaST (CC BY-NC 4.0) (Ye et al., 2022), KsponSpeech (MIT License) (Bang et al., 2020), LibriSpeech (CC BY 4.0) (Panayotov et al., 2015), Multilingual LibriSpeech (CC BY 4.0) (Pratap et al., 2020), MagicData (CC BY-NC-ND 4.0)11, MuST-C (CC
8https://www.openslr.org/62/ 9https://www.iarpa.gov/research-programs/
babel 10Resources 32, 35, 36, 37, 41, 42, 43, 44, 52, 53, 54, 61, 63, 64, 65, 66, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, and 86 from openslr.org.
11https://openslr.org/68/
BY NC ND 4.0 International) (Cattoni et al., 2021), SPGISpeech (O’Neill et al., 2021), TEDLIUM3 (CC BY-NC-ND 3.0) (Hernandez et al., 2018), ReazonSpeech (Apache 2.0 / CDLA-Sharing-1.0) (Yin et al., 2023), Russian OpenSTT (CC-BY-NC)12, VCTK (CC BY 4.0)13, VoxForge (GPL)14, VoxPopuli (Attribution-NonCommercial 4.0 International) (Wang et al., 2021), WenetSpeech (Creative Commons Attribution 4.0 International License) (Zhang et al., 2022).
A.2 Model architectures
Table 12 shows the model configurations. Our OWSM-CTC mostly follows the design of OWSM v3.1 medium (Peng et al., 2024), but we only use an encoder. To match the total model size, we increase the number of layers to 27, leading to a total of 1B parameters. Note that the sequence length of the encoder is usually longer than that of
12https://github.com/snakers4/open_stt 13https://huggingface.co/datasets/vctk 14https://www.voxforge.org/
the decoder. Hence, the encoder-only model can have a higher computational cost than the encoderdecoder model. To alleviate this issue, we apply a larger downsampling rate in the CNN module to reduce the sequence length. Our final time shift is 80ms, as opposed to 40ms of the encoderdecoder OWSM models. We observe that our training time for a fixed number of updates is roughly the same as that of OWSM v3.1 medium. We also investigated different downsampling strategies at a smaller scale, as discussed in Appendix B.1 and Table 14.
A.3 Training hyperparameters
Table 13 presents the training hyperparameters of OWSM v3.1 and our OWSM-CTC. Again, we follow the previous OWSM v3.1 (Peng et al., 2024) for a fair comparison, except that we adopt selfconditioned CTC (Nozaki and Komatsu, 2021) at
four intermediate layers (see Section 3.2).

heading: B Small-Scale Ablation Studies
text: Before the large-scale training using the entire 180k hours of audio data, we conducted preliminary experiments on MuST-C v2 En-De (Cattoni et al., 2021) to investigate the effect of the CNN downsampling rate and the choice of the task for intermediate CTC layers. Specifically, we train 24-layer EBranchformer-CTC models on the combined ASR and ST data from MuST-C v2 En-De. The input is always English audio, but the output can be the English ASR transcript or its German translation depending on the task specifier (see Figure 2).
B.1 Effect of downsampling strategies
Table 14 compares different downsampling strategies while the other configurations are kept the same. The attention is implemented with FlashAt-
tention (Dao et al., 2022). Self-conditioned CTC is applied at three intermediate layers: 6, 12, and 18. The first two CTC layers always perform ASR, while the others are task-dependent. The results show that using 8x downsampling in the CNN module leads to a slight degradation on WER and BLEU but reduces the GPU memory usage by half. We thus decide to employ 8x downsampling in our large-scale OWSM-CTC, enabling a doubled batch size per GPU. As mentioned in Appendix A.2, with this strategy, we observe a similar training speed compared to the encoder-decoder OWSM model.
B.2 Choice of the CTC task
As discussed in Section 3.2, the intermediate CTC layers can be configured to perform a specific task like ASR or multiple tasks depending on the input task token. Table 15 compares different choices at a small scale using MuST-C v2 En-De. If all CTC layers are task-dependent (i.e., multi-tasking), the model cannot converge when trained from scratch. As more layers are used for ASR only, the ASR WER improves, but the ST BLEU decreases slightly. A good trade-off is to use the first half
for ASR only and the second half for multi-tasking. Therefore, in our large-scale OWSM-CTC with 27 layers, we configure the 6th, 12th, and 15th layers to perform ASR only and the other two CTC layers (i.e., 21st and 27th layers) to be multi-tasking. This design also mimics the conventional cascaded system for ST.

heading: C More Results of ST
text: Section 4.4 shows the BLEU scores using true case with punctuation. In this section, Table 16 and Table 17 present BLEU in lowercase without punctuation, which is consistent with the setup in prior work (Peng et al., 2024). The findings are very consistent with those in Section 4.4. Our OWSM-CTC achieves higher BLEU scores with faster inference speeds than the encoder-decoder OWSM v3.1 in general.

heading: D Effect of text prompt
text: Table 18 presents an example from TEDLIUM, where the text prompt changes the output style. When there is no prompt, the ASR output of OWSM-CTC is in true case with punctuation, and
the apostrophes are combined with the previous words. However, when the previous sentence is used as a prompt, the style of the ASR hypothesis becomes more similar to that of the prompt. Specifically, the text is now in lowercase without punctuation marks, and the apostrophes are separate from previous words. This style is closer to the groundtruth transcript.
Although the above example looks promising for biasing the model’s output toward certain directions, we note that this is not guaranteed to work in a zero-shot manner. We have also tried a few examples for zero-shot contextual biasing, where we provide a few biasing words in the prompt (e.g., person names), but we find that the model may not generate the correct word in many cases. This is mainly because the model is not really trained to perform this type of task - we just provide the previous sentence (according to some probability) as the prompt during training, which might not be
useful at all; thus, the non-autoregressive model can simply ignore it in most cases. A more practical way to utilize this feature is to fine-tune our pre-trained model using some carefully designed data for contextual biasing. We will explore this in the future.

heading: E Robustness
text: Table 19 shows that autoregressive decoding sometimes fails to generate the correct output for either ASR or ST, while non-autoregressive decoding is generally more robust to this type of error.

heading: Future Work
text: We will explore this in the future